|
|
<html><head><title>pcre2serialize specification</title></head><body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"><h1>pcre2serialize man page</h1><p>Return to the <a href="index.html">PCRE2 index page</a>.</p><p>This page is part of the PCRE2 HTML documentation. It was generatedautomatically from the original man page. If there is any nonsense in it,please consult the man page, in case the conversion went wrong.<br><ul><li><a name="TOC1" href="#SEC1">SAVING AND RE-USING PRECOMPILED PCRE2 PATTERNS</a><li><a name="TOC2" href="#SEC2">SECURITY CONCERNS</a><li><a name="TOC3" href="#SEC3">SAVING COMPILED PATTERNS</a><li><a name="TOC4" href="#SEC4">RE-USING PRECOMPILED PATTERNS</a><li><a name="TOC5" href="#SEC5">AUTHOR</a><li><a name="TOC6" href="#SEC6">REVISION</a></ul><br><a name="SEC1" href="#TOC1">SAVING AND RE-USING PRECOMPILED PCRE2 PATTERNS</a><br><P><b>int32_t pcre2_serialize_decode(pcre2_code **<i>codes</i>,</b><b> int32_t <i>number_of_codes</i>, const uint8_t *<i>bytes</i>,</b><b> pcre2_general_context *<i>gcontext</i>);</b><br><br><b>int32_t pcre2_serialize_encode(const pcre2_code **<i>codes</i>,</b><b> int32_t <i>number_of_codes</i>, uint8_t **<i>serialized_bytes</i>,</b><b> PCRE2_SIZE *<i>serialized_size</i>, pcre2_general_context *<i>gcontext</i>);</b><br><br><b>void pcre2_serialize_free(uint8_t *<i>bytes</i>);</b><br><br><b>int32_t pcre2_serialize_get_number_of_codes(const uint8_t *<i>bytes</i>);</b><br><br>If you are running an application that uses a large number of regularexpression patterns, it may be useful to store them in a precompiled forminstead of having to compile them every time the application is run. However,if you are using the just-in-time optimization feature, it is not possible tosave and reload the JIT data, because it is position-dependent. The host onwhich the patterns are reloaded must be running the same version of PCRE2, withthe same code unit width, and must also have the same endianness, pointer widthand PCRE2_SIZE type. For example, patterns compiled on a 32-bit system usingPCRE2's 16-bit library cannot be reloaded on a 64-bit system, nor can they bereloaded using the 8-bit library.</P><P>Note that "serialization" in PCRE2 does not convert compiled patterns to anabstract format like Java or .NET serialization. The serialized output isreally just a bytecode dump, which is why it can only be reloaded in the sameenvironment as the one that created it. Hence the restrictions mentioned above.Applications that are not statically linked with a fixed version of PCRE2 mustbe prepared to recompile patterns from their sources, in order to be immune toPCRE2 upgrades.</P><br><a name="SEC2" href="#TOC1">SECURITY CONCERNS</a><br><P>The facility for saving and restoring compiled patterns is intended for usewithin individual applications. As such, the data supplied to<b>pcre2_serialize_decode()</b> is expected to be trusted data, not data fromarbitrary external sources. There is only some simple consistency checking, notcomplete validation of what is being re-loaded. Corrupted data may causeundefined results. For example, if the length field of a pattern in theserialized data is corrupted, the deserializing code may read beyond the end ofthe byte stream that is passed to it.</P><br><a name="SEC3" href="#TOC1">SAVING COMPILED PATTERNS</a><br><P>Before compiled patterns can be saved they must be serialized, which in PCRE2means converting the pattern to a stream of bytes. A single byte stream maycontain any number of compiled patterns, but they must all use the samecharacter tables. A single copy of the tables is included in the byte stream(its size is 1088 bytes). For more details of character tables, see the<a href="pcre2api.html#localesupport">section on locale support</a>in the<a href="pcre2api.html"><b>pcre2api</b></a>documentation.</P><P>The function <b>pcre2_serialize_encode()</b> creates a serialized byte streamfrom a list of compiled patterns. Its first two arguments specify the list,being a pointer to a vector of pointers to compiled patterns, and the length ofthe vector. The third and fourth arguments point to variables which are set topoint to the created byte stream and its length, respectively. The finalargument is a pointer to a general context, which can be used to specify custommemory management functions. If this argument is NULL, <b>malloc()</b> is usedto obtain memory for the byte stream. The yield of the function is the numberof serialized patterns, or one of the following negative error codes:<pre> PCRE2_ERROR_BADDATA the number of patterns is zero or less PCRE2_ERROR_BADMAGIC mismatch of id bytes in one of the patterns PCRE2_ERROR_NOMEMORY memory allocation failed PCRE2_ERROR_MIXEDTABLES the patterns do not all use the same tables PCRE2_ERROR_NULL the 1st, 3rd, or 4th argument is NULL</pre>PCRE2_ERROR_BADMAGIC means either that a pattern's code has been corrupted, orthat a slot in the vector does not point to a compiled pattern.</P><P>Once a set of patterns has been serialized you can save the data in anyappropriate manner. Here is sample code that compiles two patterns and writesthem to a file. It assumes that the variable <i>fd</i> refers to a file that isopen for output. The error checking that should be present in a realapplication has been omitted for simplicity.<pre> int errorcode; uint8_t *bytes; PCRE2_SIZE erroroffset; PCRE2_SIZE bytescount; pcre2_code *list_of_codes[2]; list_of_codes[0] = pcre2_compile("first pattern", PCRE2_ZERO_TERMINATED, 0, &errorcode, &erroroffset, NULL); list_of_codes[1] = pcre2_compile("second pattern", PCRE2_ZERO_TERMINATED, 0, &errorcode, &erroroffset, NULL); errorcode = pcre2_serialize_encode(list_of_codes, 2, &bytes, &bytescount, NULL); errorcode = fwrite(bytes, 1, bytescount, fd);</pre>Note that the serialized data is binary data that may contain any of the 256possible byte values. On systems that make a distinction between binary andnon-binary data, be sure that the file is opened for binary output.</P><P>Serializing a set of patterns leaves the original data untouched, so they canstill be used for matching. Their memory must eventually be freed in the usualway by calling <b>pcre2_code_free()</b>. When you have finished with the bytestream, it too must be freed by calling <b>pcre2_serialize_free()</b>. If thisfunction is called with a NULL argument, it returns immediately without doinganything.</P><br><a name="SEC4" href="#TOC1">RE-USING PRECOMPILED PATTERNS</a><br><P>In order to re-use a set of saved patterns you must first make the serializedbyte stream available in main memory (for example, by reading from a file). Themanagement of this memory block is up to the application. You can use the<b>pcre2_serialize_get_number_of_codes()</b> function to find out how manycompiled patterns are in the serialized data without actually decoding thepatterns:<pre> uint8_t *bytes = <serialized data>; int32_t number_of_codes = pcre2_serialize_get_number_of_codes(bytes);</pre>The <b>pcre2_serialize_decode()</b> function reads a byte stream and recreatesthe compiled patterns in new memory blocks, setting pointers to them in avector. The first two arguments are a pointer to a suitable vector and itslength, and the third argument points to a byte stream. The final argument is apointer to a general context, which can be used to specify custom memorymanagement functions for the decoded patterns. If this argument is NULL,<b>malloc()</b> and <b>free()</b> are used. After deserialization, the bytestream is no longer needed and can be discarded.<pre> pcre2_code *list_of_codes[2]; uint8_t *bytes = <serialized data>; int32_t number_of_codes = pcre2_serialize_decode(list_of_codes, 2, bytes, NULL);</pre>If the vector is not large enough for all the patterns in the byte stream, itis filled with those that fit, and the remainder are ignored. The yield of thefunction is the number of decoded patterns, or one of the following negativeerror codes:<pre> PCRE2_ERROR_BADDATA second argument is zero or less PCRE2_ERROR_BADMAGIC mismatch of id bytes in the data PCRE2_ERROR_BADMODE mismatch of code unit size or PCRE2 version PCRE2_ERROR_BADSERIALIZEDDATA other sanity check failure PCRE2_ERROR_MEMORY memory allocation failed PCRE2_ERROR_NULL first or third argument is NULL</pre>PCRE2_ERROR_BADMAGIC may mean that the data is corrupt, or that it was compiledon a system with different endianness.</P><P>Decoded patterns can be used for matching in the usual way, and must be freedby calling <b>pcre2_code_free()</b>. However, be aware that there is a potentialrace issue if you are using multiple patterns that were decoded from a singlebyte stream in a multithreaded application. A single copy of the charactertables is used by all the decoded patterns and a reference count is used toarrange for its memory to be automatically freed when the last pattern isfreed, but there is no locking on this reference count. Therefore, if you wantto call <b>pcre2_code_free()</b> for these patterns in different threads, youmust arrange your own locking, and ensure that <b>pcre2_code_free()</b> cannotbe called by two threads at the same time.</P><P>If a pattern was processed by <b>pcre2_jit_compile()</b> before beingserialized, the JIT data is discarded and so is no longer available after asave/restore cycle. You can, however, process a restored pattern with<b>pcre2_jit_compile()</b> if you wish.</P><br><a name="SEC5" href="#TOC1">AUTHOR</a><br><P>Philip Hazel<br>Retired from University Computing Service<br>Cambridge, England.<br></P><br><a name="SEC6" href="#TOC1">REVISION</a><br><P>Last updated: 27 June 2018<br>Copyright © 1997-2018 University of Cambridge.<br><p>Return to the <a href="index.html">PCRE2 index page</a>.</p>
|
