Boost C++ Libraries

...one of the most highly regarded and expertly designed C++ library projects in the world. Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

PrevUpHomeNext

Containers in managed memory segments

Container requirements for Boost.Interprocess allocators
Portable containers in managed memory segments
Where is this being allocated?
Containers of containers

Boost.Interprocess STL compatible allocators offer a STL compatible allocator interface and if they define their internal pointer typedef as a relative pointer, they can be used to place STL containers in shared memory, memory mapped files or in a user defined memory segment.

Originally, in C++98 and C++03, this pointer typedef was useless. As Scott Meyers mentions in his Effective STL book, Item 10, "Be aware of allocator conventions and restrictions":

  • "the Standard explicitly allows library implementers to assume that every allocator's pointer typedef is a synonym for T*"
  • "the Standard says that an implementation of the STL is permitted to assume that all allocator objects of the same type are equivalent and always compare equal"

Obviously, if any STL implementation ignores pointer typedefs, no smart pointer can be used as allocator::pointer. If STL implementations assume all allocator objects of the same type compare equal, it will assume that two allocators, each one allocating from a different memory pool are equal, which is a complete disaster.

STL containers that we want to place in shared memory or memory mapped files with Boost.Interprocess can't make any of these assumptions, so:

  • STL containers may not assume that memory allocated with an allocator can be deallocated with other allocators of the same type. All allocators objects must compare equal only if memory allocated with one object can be deallocated with the other one, and this can only tested with operator==() at run-time.
  • Containers' internal pointers should be of the type allocator::pointer and containers may not assume allocator::pointer is a raw pointer.

This was in theory fixed in C++11, when std::pointer_traits utilities were introduced and standard library containers added support for them. However, due to ABI concerns and low user demand, some implementations still don't fully support fancy pointer types like boost::interprocess:offset_ptr. Those implementations still use raw pointers in some containers for internal data or iterator implementations.

This non-portable situation is described in "P0773R0: Towards meaningful fancy pointers"

Can we use offset fancy pointer types to enable safe sharing of memory regions?

Yes; but the requirements on vendors are unclear, and there are pitfalls for the programmer.

Scenario (A), offset pointers, has only one tenable solution, "continue to partly support."

[Important] Important

Since Boost.Container was created from Boost.Interprocess in 2011, this library has mantained several <boost/interprocess/containers/.hpp> headers for backwards compatibility. Those headers are now deprecated and will be removed in a future Boost version. Users should directly use Boost.Container headers

Due to the described partial support from Standard Library implementations Boost.Interprocess highly recommends using Boost.Container containers, which are guaranteed to work with Boost.Interprocess

The following Boost.Container containers are compatible with Boost.Interprocess:

  • deque
  • devector
  • flat_map/multimap
  • flat_set/multiset
  • list
  • map/multimap
  • set/multiset
  • slist
  • small_vector
  • stable_vector
  • static_vector
  • string
  • vector

To place any of these containers in managed memory segments, we must define the allocator template parameter with a Boost.Interprocess allocator so that the container allocates the values in the managed memory segment. To place the container itself in shared memory, we construct it in the managed memory segment just like any other object with Boost.Interprocess:

#include <boost/container/vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>

int main ()
{
   using namespace boost::interprocess;
   //Remove shared memory on construction and destruction
   struct shm_remove
   {
      shm_remove() { shared_memory_object::remove("MyName"); }
      ~shm_remove(){ shared_memory_object::remove("MyName"); }
   } remover;

   //A managed shared memory where we can construct objects
   //associated with a c-string
   managed_shared_memory segment(create_only,"MyName", 65536);

   //Alias an STL-like allocator of ints that allocates ints from the segment
   typedef allocator<int, managed_shared_memory::segment_manager>
      ShmemAllocator;

   //Alias a vector that uses the previous STL-like allocator
   typedef boost::container::vector<int, ShmemAllocator> MyVector;

   int initVal[]        = {0, 1, 2, 3, 4, 5, 6 };
   const int *begVal    = initVal;
   const int *endVal    = initVal + sizeof(initVal)/sizeof(initVal[0]);

   //Initialize the STL-like allocator
   const ShmemAllocator alloc_inst (segment.get_segment_manager());

   //Construct the vector in the shared memory segment with the STL-like allocator
   //from a range of iterators
   MyVector *myvector =
      segment.construct<MyVector>
         ("MyVector")/*object name*/
         (begVal     /*first ctor parameter*/
         ,endVal     /*second ctor parameter*/
         ,alloc_inst /*third ctor parameter*/);

   //Use vector as your want
   std::sort(myvector->rbegin(), myvector->rend());
   // . . .
   //When done, destroy and delete vector from the segment
   segment.destroy<MyVector>("MyVector");
   return 0;
}

These containers also show how easy is to create/modify an existing container making possible to place it in shared memory.

Boost.Interprocess containers are placed in shared memory/memory mapped files, etc... using two mechanisms at the same time:

  • Boost.Interprocess construct<>, find_or_construct<>... functions. These functions place a C++ object in the shared memory/memory mapped file. But this places only the object, but not the memory that this object may allocate dynamically.
  • Shared memory allocators. These allow allocating shared memory/memory mapped file portions so that containers can allocate dynamically fragments of memory to store newly inserted elements.

This means that to place any Boost.Interprocess container (including Boost.Interprocess strings) in shared memory or memory mapped files, containers must:

  • Define their template allocator parameter to a Boost.Interprocess allocator.
  • Every container constructor must take the Boost.Interprocess allocator as parameter.
  • You must use construct<>/find_or_construct<>... functions to place the container in the managed memory.

If you do the first two points but you don't use construct<> or find_or_construct<> you are creating a container placed only in your process but that allocates memory for contained types from shared memory/memory mapped file.

Let's see an example:

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/container/vector.hpp>
#include <boost/container/string.hpp>
#include <boost/interprocess/allocators/allocator.hpp>

int main ()
{
   using namespace boost::interprocess;
   //Typedefs
   typedef allocator<char, managed_shared_memory::segment_manager>
      CharAllocator;
   typedef boost::container::basic_string<char, std::char_traits<char>, CharAllocator>
      MyShmString;
   typedef allocator<MyShmString, managed_shared_memory::segment_manager>
      StringAllocator;
   typedef boost::container::vector<MyShmString, StringAllocator>
      MyShmStringVector;

   //Open shared memory
   //Remove shared memory on construction and destruction
   struct shm_remove
   {
      shm_remove() { shared_memory_object::remove("MyName"); }
      ~shm_remove(){ shared_memory_object::remove("MyName"); }
   } remover;

   managed_shared_memory shm(create_only, "MyName", 10000);

   //Create allocators
   CharAllocator     charallocator  (shm.get_segment_manager());
   StringAllocator   stringallocator(shm.get_segment_manager());

   //This string is in only in this process (the pointer pointing to the
   //buffer that will hold the text is not in shared memory).
   //But the buffer that will hold "this is my text" is allocated from
   //shared memory
   MyShmString mystring(charallocator);
   mystring = "this is my text";

   //This vector is only in this process (the pointer pointing to the
   //buffer that will hold the MyShmString-s is not in shared memory).
   //But the buffer that will hold 10 MyShmString-s is allocated from
   //shared memory using StringAllocator. Since strings use a shared
   //memory allocator (CharAllocator) the 10 buffers that hold
   //"this is my text" text are also in shared memory.
   MyShmStringVector myvector(stringallocator);
   myvector.insert(myvector.begin(), 10, mystring);

   //This vector is fully constructed in shared memory. All pointers
   //buffers are constructed in the same shared memory segment
   //This vector can be safely accessed from other processes.
   MyShmStringVector *myshmvector =
      shm.construct<MyShmStringVector>("myshmvector")(stringallocator);
   myshmvector->insert(myshmvector->begin(), 10, mystring);

   //Destroy vector. This will free all strings that the vector contains
   shm.destroy_ptr(myshmvector);
   return 0;
}

When creating containers of containers, each container needs an allocator. To avoid using several allocators with complex type definitions, we can take advantage of the type erasure provided by void allocators and the ability to implicitly convert void allocators in allocators that allocate other types.

Here we have an example that builds a map in shared memory. Key is a string and the mapped type is a class that stores several containers:

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/container/map.hpp>
#include <boost/container/vector.hpp>
#include <boost/container/string.hpp>

using namespace boost::interprocess;

//Typedefs of allocators and containers
typedef managed_shared_memory::segment_manager                       segment_manager_t;
typedef allocator<void, segment_manager_t>                           void_allocator;
typedef allocator<int, segment_manager_t>                            int_allocator;
typedef boost::container::vector<int, int_allocator>                 int_vector;
typedef allocator<int_vector, segment_manager_t>                     int_vector_allocator;
typedef boost::container::vector<int_vector, int_vector_allocator>   int_vector_vector;
typedef allocator<char, segment_manager_t>                           char_allocator;
typedef boost::container::basic_string<char, std::char_traits<char>, char_allocator>   char_string;

class complex_data
{
   int               id_;
   char_string       char_string_;
   int_vector_vector int_vector_vector_;

   public:
   //Since void_allocator is convertible to any other allocator<T>, we can simplify
   //the initialization taking just one allocator for all inner containers.
   complex_data(int id, const char *name, const void_allocator &void_alloc)
      : id_(id), char_string_(name, void_alloc), int_vector_vector_(void_alloc)
   {}
   //Other members...
};

//Definition of the map holding a string as key and complex_data as mapped type
typedef std::pair<const char_string, complex_data>                      map_value_type;
typedef std::pair<char_string, complex_data>                            movable_to_map_value_type;
typedef allocator<map_value_type, segment_manager_t>                    map_value_type_allocator;
typedef boost::container::map< char_string, complex_data
           , std::less<char_string>, map_value_type_allocator>          complex_map_type;

int main ()
{
   //Remove shared memory on construction and destruction
   struct shm_remove
   {
      shm_remove() { shared_memory_object::remove("MyName"); }
      ~shm_remove(){ shared_memory_object::remove("MyName"); }
   } remover;

   //Create shared memory
   managed_shared_memory segment(create_only,"MyName", 65536);

   //An allocator convertible to any allocator<T, segment_manager_t> type
   void_allocator alloc_inst (segment.get_segment_manager());

   //Construct the shared memory map and fill it
   complex_map_type *mymap = segment.construct<complex_map_type>
      //(object name), (first ctor parameter, second ctor parameter)
         ("MyMap")(std::less<char_string>(), alloc_inst);

   for(int i = 0; i < 100; ++i){
      //Both key(string) and value(complex_data) need an allocator in their constructors
      char_string  key_object(alloc_inst);
      complex_data mapped_object(i, "default_name", alloc_inst);
      map_value_type value(key_object, mapped_object);
      //Modify values and insert them in the map
      mymap->insert(value);
   }
   return 0;
}

PrevUpHomeNext