sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-09-22 17:16:19 -07:00
Code Issues Packages Projects Releases Wiki Activity

feat(dictionary): add Hachidori backend support

Browse Source 
- Add backend selection, setup gating, Anki integration, and external host support
- Add launcher flags, documentation, packaging, and focused tests
- Open on-demand overlay modals on the first attempt
This commit is contained in:
sudacode
2026-09-22 00:21:19 -07:00
parent 1508863dbb
commit d9fdc7ef6d
446 changed files with 109060 additions and 244 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/LICENSE
Vendored
+179
View File
@@ -0,0 +1,179 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/LICENSE-hashmap.c
Vendored
+24
View File
@@ -0,0 +1,24 @@
hashmap.c is MIT, Copyright (c) 2020 Joshua J Baker (https://github.com/tidwall/hashmap.c):
The MIT License (MIT)
Copyright (c) 2020 Joshua J Baker
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/README.md
Vendored
+17
View File
@@ -0,0 +1,17 @@
# gumbo-parser (vendored)
HTML5 parser used by the MDX importer to turn MDX glossary HTML into Yomitan
structured content.
Source: the `gumbo-parser/src` tree of Nokogiri
(https://github.com/sparklemotion/nokogiri, commit
da64104acfdc8b595e49935499c9cc357ebcac7a), itself a maintained fork of
google/gumbo-parser via lua-gumbo. Only the library sources (`src/*.c`,
`src/*.h`) are copied; Nokogiri's tests, fuzzers and the gperf/ragel inputs
are not. No file is modified. See UPSTREAM-README.md for the fork's history.
Licence: Apache-2.0 (LICENSE); `src/hashmap.c` is MIT (LICENSE-hashmap.c).
To update: copy `gumbo-parser/src/*.{c,h}` and `gumbo-parser/src/README.md`
(as UPSTREAM-README.md) from a newer Nokogiri checkout and bump the commit
above.
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/UPSTREAM-README.md
Vendored
+41
View File
@@ -0,0 +1,41 @@
libgumbo
========
This is an internal fork of the [libgumbo] library, which was copied and
later modified under the terms of the Apache 2.0 [license]. See `lua-gumbo`
commit [`0a04728`] for details of the original import.
Since importing the code, the following notable fixes and improvements
have been made:
* `91cef89`: Re-implement `adjust_foreign_attributes()` with a gperf hash
* `b11abe7`: Pass `TagSet` arrays into functions by reference instead of value
* `b73dc03`: Simplify `maybe_replace_codepoint()` function
* `d5d0bb3`: Remove special handling of `<menuitem>` tag
* `7bd5162`: Remove special handling of `<isindex>` tag
* `a5c1b0e`: Use `realloc(3)` instead of `malloc(3)` in `enlarge_vector_if_full()`
* `dcbebd7`: Use `realloc(3)` instead of `malloc(3)` in `maybe_resize_string_buffer()`
* `df15262`: Make `destroy_node()` function non-recursive
* `2df37f5`: Fix signedness of some format specifiers
* `176553e`: Add maximum element nesting limit
* `bed0f4a`: Annotate `gumbo_debug()` with `PRINTF` macro and fix warnings
* `7ffc218`: Annotate `print_message()` with `PRINTF` macro and fix warnings
* `1bd8ab5`, `9136507`, `53a1f9a`: Deduplicate some identical `TagSet` arrays
* `a7a9065`: Add some GCC/Clang function attributes
* `8d3d4e4`: Remove custom allocator support
* `8d3b006`: Fix recording of source positions for `</form>` end tags
* `1a8d763`: Replace linear search in `maybe_replace_codepoint()` with a lookup table
* `6dca79e`: Replace `strcasecmp()` and `strncasecmp()` with ascii-only equivalents
* `17ab1d2`: Fix `TAGSET_INCLUDES` macro to work properly with multiple bit flags
* `7e56d45`: Re-implement `gumbo_normalize_svg_tagname()` with a gperf hash
* `a518d35`: Replace linear array search in `adjust_svg_attributes()` with a gperf hash
* `a4a7433`: Fix duplicate `TagSet` initializer being ignored in `is_special_node()`
* `8137fcd`: Add support for `<dialog>` tag
* `4b35471`: Add missing `static` qualifiers to hide symbols that shouldn't be extern
* `df57c59`, `03101f3`, `ea62330`: Replace use of locale-dependant `ctype.h` functions
with custom, ASCII-only equivalents
[libgumbo]: https://github.com/google/gumbo-parser/tree/aa91b27b02c0c80c482e24348a457ed7c3c088e0/src
[license]: https://github.com/google/gumbo-parser/blob/aa91b27b02c0c80c482e24348a457ed7c3c088e0/COPYING
[`0a04728`]: https://gitlab.com/craigbarnes/lua-gumbo/commit/0a047282815af86f3367a7d95fefcfe5723ece48
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/ascii.c
Vendored
+75
View File
@@ -0,0 +1,75 @@
#include "ascii.h"
int gumbo_ascii_strcasecmp(const char *s1, const char *s2) {
int c1, c2;
while (*s1 && *s2) {
c1 = (int)(unsigned char) gumbo_ascii_tolower(*s1);
c2 = (int)(unsigned char) gumbo_ascii_tolower(*s2);
if (c1 != c2) {
return (c1 - c2);
}
s1++;
s2++;
}
return (((int)(unsigned char) *s1) - ((int)(unsigned char) *s2));
}
int gumbo_ascii_strncasecmp(const char *s1, const char *s2, size_t n) {
int c1, c2;
while (n && *s1 && *s2) {
n -= 1;
c1 = (int)(unsigned char) gumbo_ascii_tolower(*s1);
c2 = (int)(unsigned char) gumbo_ascii_tolower(*s2);
if (c1 != c2) {
return (c1 - c2);
}
s1++;
s2++;
}
if (n) {
return (((int)(unsigned char) *s1) - ((int)(unsigned char) *s2));
}
return 0;
}
const unsigned char _gumbo_ascii_table[0x80] = {
0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x03,0x03,0x01,0x03,0x03,0x01,0x01,
0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,0x01,
0x02,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x00,0x00,0x00,0x00,0x00,0x00,
0x00,0x28,0x28,0x28,0x28,0x28,0x28,0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x20,
0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x20,0x00,0x00,0x00,0x00,0x00,
0x00,0x50,0x50,0x50,0x50,0x50,0x50,0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x40,
0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x40,0x00,0x00,0x00,0x00,0x00,
};
// Table generation code.
// clang -DGUMBO_GEN_TABLE ascii.c && ./a.out && rm a.out
#ifdef GUMBO_GEN_TABLE
#include <stdio.h>
int main() {
printf("const unsigned char _gumbo_ascii_table[0x80] = {");
for (int c = 0; c < 0x80; ++c) {
unsigned int x = 0;
// https://infra.spec.whatwg.org/#ascii-code-point
if (c <= 0x1f)
x |= GUMBO_ASCII_CNTRL;
if (c == 0x09 || c == 0x0a || c == 0x0c || c == 0x0d || c == 0x20)
x |= GUMBO_ASCII_SPACE;
if (c >= 0x30 && c <= 0x39)
x |= GUMBO_ASCII_DIGIT;
if ((c >= 0x30 && c <= 0x39) || (c >= 0x41 && c <= 0x46))
x |= GUMBO_ASCII_UPPER_XDIGIT;
if ((c >= 0x30 && c <= 0x39) || (c >= 0x61 && c <= 0x66))
x |= GUMBO_ASCII_LOWER_XDIGIT;
if (c >= 0x41 && c <= 0x5a)
x |= GUMBO_ASCII_UPPER_ALPHA;
if (c >= 0x61 && c <= 0x7a)
x |= GUMBO_ASCII_LOWER_ALPHA;
printf("%s0x%02x,", (c % 16 == 0? "\n " : ""), x);
}
printf("\n};\n");
return 0;
}
#endif
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/ascii.h
Vendored
+115
View File
@@ -0,0 +1,115 @@
#ifndef GUMBO_ASCII_H_
#define GUMBO_ASCII_H_
#include <stddef.h>
#include "macros.h"
#ifdef __cplusplus
extern "C" {
#endif
PURE NONNULL_ARGS
int gumbo_ascii_strcasecmp(const char *s1, const char *s2);
PURE NONNULL_ARGS
int gumbo_ascii_strncasecmp(const char *s1, const char *s2, size_t n);
// If these values change, then _gumbo_ascii_table needs to be regenerated.
#define GUMBO_ASCII_CNTRL 1
#define GUMBO_ASCII_SPACE 2
#define GUMBO_ASCII_DIGIT 4
#define GUMBO_ASCII_UPPER_XDIGIT 8
#define GUMBO_ASCII_LOWER_XDIGIT 16
#define GUMBO_ASCII_UPPER_ALPHA 32
#define GUMBO_ASCII_LOWER_ALPHA 64
#define GUMBO_ASCII_XDIGIT (GUMBO_ASCII_LOWER_XDIGIT | GUMBO_ASCII_UPPER_XDIGIT)
#define GUMBO_ASCII_ALPHA (GUMBO_ASCII_UPPER_ALPHA | GUMBO_ASCII_LOWER_ALPHA)
#define GUMBO_ASCII_ALNUM (GUMBO_ASCII_DIGIT | GUMBO_ASCII_ALPHA)
extern const unsigned char _gumbo_ascii_table[0x80];
CONST_FN
static inline int gumbo_ascii_isascii(int c) {
return ((unsigned int)c & ~0x7fu) == 0;
}
// 0x00 -- 0x1F (A C0 control)
CONST_FN
static inline int gumbo_ascii_iscntrl(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_CNTRL);
}
// 0x09, 0x0a, 0x0c, 0x0d, 0x20
CONST_FN
static inline int gumbo_ascii_isspace(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_SPACE);
}
CONST_FN
static inline int gumbo_ascii_istab_or_newline(int c) {
return c == 0x09 || c == 0x0a || c == 0x0d;
}
CONST_FN
static inline int gumbo_ascii_isdigit(int c) {
return c >= 0x30 && c <= 0x39;
}
CONST_FN
static inline int gumbo_ascii_isalpha(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_ALPHA);
}
CONST_FN
static inline int gumbo_ascii_isxdigit(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_XDIGIT);
}
CONST_FN
static inline int gumbo_ascii_isupper_xdigit(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_UPPER_XDIGIT);
}
CONST_FN
static inline int gumbo_ascii_islower_xdigit(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_LOWER_XDIGIT);
}
CONST_FN
static inline int gumbo_ascii_isupper(int c) {
return ((unsigned)(c) - 'A') < 26;
}
CONST_FN
static inline int gumbo_ascii_islower(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_LOWER_ALPHA);
}
CONST_FN
static inline int gumbo_ascii_isalnum(int c) {
return gumbo_ascii_isascii(c)
&& (_gumbo_ascii_table[c] & GUMBO_ASCII_ALNUM);
}
CONST_FN
static inline int gumbo_ascii_tolower(int c) {
if (gumbo_ascii_isupper(c)) {
return c | 32;
}
return c;
}
#ifdef __cplusplus
}
#endif
#endif // GUMBO_ASCII_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/attribute.c
Vendored
+42
View File
@@ -0,0 +1,42 @@
/*
Copyright 2018 Craig Barnes.
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include "attribute.h"
#include "ascii.h"
#include "util.h"
GumboAttribute* gumbo_get_attribute (
const GumboVector* attributes,
const char* name
) {
for (unsigned int i = 0; i < attributes->length; ++i) {
GumboAttribute* attr = attributes->data[i];
if (!gumbo_ascii_strcasecmp(attr->name, name)) {
return attr;
}
}
return NULL;
}
void gumbo_destroy_attribute(GumboAttribute* attribute) {
gumbo_free((void*) attribute->name);
gumbo_free((void*) attribute->value);
gumbo_free((void*) attribute);
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/attribute.h
Vendored
+17
View File
@@ -0,0 +1,17 @@
#ifndef GUMBO_ATTRIBUTE_H_
#define GUMBO_ATTRIBUTE_H_
#include "nokogiri_gumbo.h"
#ifdef __cplusplus
extern "C" {
#endif
// Release the memory used for a GumboAttribute, including the attribute itself
void gumbo_destroy_attribute(GumboAttribute* attribute);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_ATTRIBUTE_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/char_ref.c
Vendored
+22225
View File
File diff suppressed because it is too large Load Diff
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/char_ref.h
Vendored
+29
View File
@@ -0,0 +1,29 @@
#ifndef GUMBO_CHAR_REF_H_
#define GUMBO_CHAR_REF_H_
#include <stdlib.h>
#ifdef __cplusplus
extern "C" {
#endif
// Value that indicates no character was produced.
#define kGumboNoChar (-1)
// On input, str points to the start of the string to match and size is the
// size of the string.
//
// Returns the length of the match or 0 if there is no match.
// output[0] contains the first codepoint and output[1] contains the second if
// there are two, otherwise output[1] contains kGumboNoChar.
size_t match_named_char_ref (
const char *str,
size_t size,
int output[2]
);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_CHAR_REF_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/error.c
Vendored
+658
View File
@@ -0,0 +1,658 @@
/*
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <assert.h>
#include <inttypes.h>
#include <stdarg.h>
#include <stdio.h>
#include <string.h>
#include "ascii.h"
#include "error.h"
#include "nokogiri_gumbo.h"
#include "macros.h"
#include "parser.h"
#include "string_buffer.h"
#include "util.h"
#include "vector.h"
// Prints a formatted message to a StringBuffer. This automatically resizes the
// StringBuffer as necessary to fit the message. Returns the number of bytes
// written.
static int PRINTF(2) print_message (
GumboStringBuffer* output,
const char* format,
...
) {
va_list args;
int remaining_capacity = output->capacity - output->length;
va_start(args, format);
int bytes_written = vsnprintf (
output->data + output->length,
remaining_capacity,
format,
args
);
va_end(args);
#if (defined(_MSC_VER) && (_MSC_VER < 1900)) || defined(_RUBY_MSVCRT)
if (bytes_written == -1) {
// vsnprintf returns -1 on older MSVC++ if there's not enough capacity,
// instead of returning the number of bytes that would've been written had
// there been enough. In this case, we can call vsnprintf() again but
// with a count of 0 to get the number of bytes written, not including
// the null terminator.
// https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/vsnprintf-vsnprintf-vsnprintf-l-vsnwprintf-vsnwprintf-l?view=msvc-140#behavior-summary
va_start(args, format);
bytes_written = vsnprintf (
NULL,
0,
format,
args
);
va_end(args);
}
#endif
// -1 in standard C99 indicates an encoding error. Return 0 and do nothing.
if (bytes_written == -1) {
return 0;
}
if (bytes_written >= remaining_capacity) {
// At least double the size of the buffer.
size_t new_capacity = output->capacity * 2;
if (new_capacity < output->length + bytes_written + 1) {
// The +1 is for the null terminator.
new_capacity = output->length + bytes_written + 1;
}
gumbo_string_buffer_reserve(new_capacity, output);
remaining_capacity = output->capacity - output->length;
va_start(args, format);
bytes_written = vsnprintf (
output->data + output->length,
remaining_capacity,
format,
args
);
va_end(args);
}
output->length += bytes_written;
return bytes_written;
}
static void print_tag_stack (
const GumboParserError* error,
GumboStringBuffer* output
) {
print_message(output, " Currently open tags: ");
for (unsigned int i = 0; i < error->tag_stack.length; ++i) {
if (i) {
print_message(output, ", ");
}
uintptr_t tag = (uintptr_t) error->tag_stack.data[i];
const char* tag_name;
if (tag > GUMBO_TAG_UNKNOWN) {
tag_name = error->tag_stack.data[i];
} else {
tag_name = gumbo_normalized_tagname((GumboTag)tag);
}
print_message(output, "%s", tag_name);
}
gumbo_string_buffer_append_codepoint('.', output);
}
static void handle_tokenizer_error (
const GumboError* error,
GumboStringBuffer* output
) {
switch (error->type) {
case GUMBO_ERR_ABRUPT_CLOSING_OF_EMPTY_COMMENT:
print_message(output, "Empty comment abruptly closed by '%s', use '-->'.",
error->v.tokenizer.state == GUMBO_LEX_COMMENT_START? ">" : "->");
break;
case GUMBO_ERR_ABRUPT_DOCTYPE_PUBLIC_IDENTIFIER:
print_message (
output,
"DOCTYPE public identifier missing closing %s.",
error->v.tokenizer.state == GUMBO_LEX_DOCTYPE_PUBLIC_ID_DOUBLE_QUOTED?
"quotation mark (\")" : "apostrophe (')"
);
break;
case GUMBO_ERR_ABRUPT_DOCTYPE_SYSTEM_IDENTIFIER:
print_message (
output,
"DOCTYPE system identifier missing closing %s.",
error->v.tokenizer.state == GUMBO_LEX_DOCTYPE_SYSTEM_ID_DOUBLE_QUOTED?
"quotation mark (\")" : "apostrophe (')"
);
break;
case GUMBO_ERR_ABSENCE_OF_DIGITS_IN_NUMERIC_CHARACTER_REFERENCE:
print_message (
output,
"Numeric character reference '%.*s' does not contain any %sdigits.",
(int)error->original_text.length, error->original_text.data,
error->v.tokenizer.state == GUMBO_LEX_HEXADECIMAL_CHARACTER_REFERENCE_START? "hexadecimal " : ""
);
break;
case GUMBO_ERR_CDATA_IN_HTML_CONTENT:
print_message(output, "CDATA section outside foreign (SVG or MathML) content.");
break;
case GUMBO_ERR_CHARACTER_REFERENCE_OUTSIDE_UNICODE_RANGE:
print_message (
output,
"Numeric character reference '%.*s' references a code point that is outside the valid Unicode range.",
(int)error->original_text.length, error->original_text.data
);
break;
case GUMBO_ERR_CONTROL_CHARACTER_IN_INPUT_STREAM:
print_message (
output,
"Input contains prohibited control code point U+%04X.",
error->v.tokenizer.codepoint
);
break;
case GUMBO_ERR_CONTROL_CHARACTER_REFERENCE:
print_message (
output,
"Numeric character reference '%.*s' references prohibited control code point U+%04X.",
(int)error->original_text.length, error->original_text.data,
error->v.tokenizer.codepoint
);
break;
case GUMBO_ERR_END_TAG_WITH_ATTRIBUTES:
print_message(output, "End tag contains attributes.");
break;
case GUMBO_ERR_DUPLICATE_ATTRIBUTE:
print_message(output, "Tag contains multiple attributes with the same name.");
break;
case GUMBO_ERR_END_TAG_WITH_TRAILING_SOLIDUS:
print_message(output, "End tag ends with '/>', use '>'.");
break;
case GUMBO_ERR_EOF_BEFORE_TAG_NAME:
print_message(output, "End of input where a tag name is expected.");
break;
case GUMBO_ERR_EOF_IN_CDATA:
print_message(output, "End of input in CDATA section.");
break;
case GUMBO_ERR_EOF_IN_COMMENT:
print_message(output, "End of input in comment.");
break;
case GUMBO_ERR_EOF_IN_DOCTYPE:
print_message(output, "End of input in DOCTYPE.");
break;
case GUMBO_ERR_EOF_IN_SCRIPT_HTML_COMMENT_LIKE_TEXT:
print_message(output, "End of input in text that resembles an HTML comment inside script element content.");
break;
case GUMBO_ERR_EOF_IN_TAG:
print_message(output, "End of input in tag.");
break;
case GUMBO_ERR_INCORRECTLY_CLOSED_COMMENT:
print_message(output, "Comment closed incorrectly by '--!>', use '-->'.");
break;
case GUMBO_ERR_INCORRECTLY_OPENED_COMMENT:
print_message(output, "Comment, DOCTYPE, or CDATA opened incorrectly, use '<!--', '<!DOCTYPE', or '<![CDATA['.");
break;
case GUMBO_ERR_INVALID_CHARACTER_SEQUENCE_AFTER_DOCTYPE_NAME:
print_message(output, "Invalid character sequence after DOCTYPE name, expected 'PUBLIC', 'SYSTEM', or '>'.");
break;
case GUMBO_ERR_INVALID_FIRST_CHARACTER_OF_TAG_NAME:
if (gumbo_ascii_isascii(error->v.tokenizer.codepoint)
&& !gumbo_ascii_iscntrl(error->v.tokenizer.codepoint))
print_message(output, "Invalid first character of tag name '%c'.", error->v.tokenizer.codepoint);
else
print_message(output, "Invalid first code point of tag name U+%04X.", error->v.tokenizer.codepoint);
break;
case GUMBO_ERR_MISSING_ATTRIBUTE_VALUE:
print_message(output, "Missing attribute value.");
break;
case GUMBO_ERR_MISSING_DOCTYPE_NAME:
print_message(output, "Missing DOCTYPE name.");
break;
case GUMBO_ERR_MISSING_DOCTYPE_PUBLIC_IDENTIFIER:
print_message(output, "Missing DOCTYPE public identifier.");
break;
case GUMBO_ERR_MISSING_DOCTYPE_SYSTEM_IDENTIFIER:
print_message(output, "Missing DOCTYPE system identifier.");
break;
case GUMBO_ERR_MISSING_END_TAG_NAME:
print_message(output, "Missing end tag name.");
break;
case GUMBO_ERR_MISSING_QUOTE_BEFORE_DOCTYPE_PUBLIC_IDENTIFIER:
print_message(output, "Missing quote before DOCTYPE public identifier.");
break;
case GUMBO_ERR_MISSING_QUOTE_BEFORE_DOCTYPE_SYSTEM_IDENTIFIER:
print_message(output, "Missing quote before DOCTYPE system identifier.");
break;
case GUMBO_ERR_MISSING_SEMICOLON_AFTER_CHARACTER_REFERENCE:
print_message(output, "Missing semicolon after character reference '%.*s'.",
(int)error->original_text.length, error->original_text.data);
break;
case GUMBO_ERR_MISSING_WHITESPACE_AFTER_DOCTYPE_PUBLIC_KEYWORD:
print_message(output, "Missing whitespace after 'PUBLIC' keyword.");
break;
case GUMBO_ERR_MISSING_WHITESPACE_AFTER_DOCTYPE_SYSTEM_KEYWORD:
print_message(output, "Missing whitespace after 'SYSTEM' keyword.");
break;
case GUMBO_ERR_MISSING_WHITESPACE_BEFORE_DOCTYPE_NAME:
print_message(output, "Missing whitespace between 'DOCTYPE' keyword and DOCTYPE name.");
break;
case GUMBO_ERR_MISSING_WHITESPACE_BETWEEN_ATTRIBUTES:
print_message(output, "Missing whitespace between attributes.");
break;
case GUMBO_ERR_MISSING_WHITESPACE_BETWEEN_DOCTYPE_PUBLIC_AND_SYSTEM_IDENTIFIERS:
print_message(output, "Missing whitespace between DOCTYPE public and system identifiers.");
break;
case GUMBO_ERR_NESTED_COMMENT:
print_message(output, "Nested comment.");
break;
case GUMBO_ERR_NONCHARACTER_CHARACTER_REFERENCE:
print_message (
output,
"Numeric character reference '%.*s' references noncharacter U+%04X.",
(int)error->original_text.length, error->original_text.data,
error->v.tokenizer.codepoint
);
break;
case GUMBO_ERR_NONCHARACTER_IN_INPUT_STREAM:
print_message(output, "Input contains noncharacter U+%04X.", error->v.tokenizer.codepoint);
break;
case GUMBO_ERR_NON_VOID_HTML_ELEMENT_START_TAG_WITH_TRAILING_SOLIDUS:
print_message(output, "Start tag of nonvoid HTML element ends with '/>', use '>'.");
break;
case GUMBO_ERR_NULL_CHARACTER_REFERENCE:
print_message(output, "Numeric character reference '%.*s' references U+0000.",
(int)error->original_text.length, error->original_text.data);
break;
case GUMBO_ERR_SURROGATE_CHARACTER_REFERENCE:
print_message (
output,
"Numeric character reference '%.*s' references surrogate U+%4X.",
(int)error->original_text.length, error->original_text.data,
error->v.tokenizer.codepoint
);
break;
case GUMBO_ERR_SURROGATE_IN_INPUT_STREAM:
print_message(output, "Input contains surrogate U+%04X.", error->v.tokenizer.codepoint);
break;
case GUMBO_ERR_UNEXPECTED_CHARACTER_AFTER_DOCTYPE_SYSTEM_IDENTIFIER:
print_message(output, "Unexpected character after DOCTYPE system identifier.");
break;
case GUMBO_ERR_UNEXPECTED_CHARACTER_IN_ATTRIBUTE_NAME:
print_message(output, "Unexpected character (%c) in attribute name.", error->v.tokenizer.codepoint);
break;
case GUMBO_ERR_UNEXPECTED_CHARACTER_IN_UNQUOTED_ATTRIBUTE_VALUE:
print_message(output, "Unexpected character (%c) in unquoted attribute value.", error->v.tokenizer.codepoint);
break;
case GUMBO_ERR_UNEXPECTED_EQUALS_SIGN_BEFORE_ATTRIBUTE_NAME:
print_message(output, "Unexpected '=' before an attribute name.");
break;
case GUMBO_ERR_UNEXPECTED_NULL_CHARACTER:
print_message(output, "Input contains unexpected U+0000.");
break;
case GUMBO_ERR_UNEXPECTED_QUESTION_MARK_INSTEAD_OF_TAG_NAME:
print_message(output, "Unexpected '?' where start tag name is expected.");
break;
case GUMBO_ERR_UNEXPECTED_SOLIDUS_IN_TAG:
print_message(output, "Unexpected '/' in tag.");
break;
case GUMBO_ERR_UNKNOWN_NAMED_CHARACTER_REFERENCE:
print_message(output, "Unknown named character reference '%.*s'.",
(int)error->original_text.length, error->original_text.data);
break;
case GUMBO_ERR_UTF8_INVALID:
print_message(output, "Invalid UTF8 encoding.");
break;
case GUMBO_ERR_UTF8_TRUNCATED:
print_message(output, "UTF8 character truncated.");
break;
case GUMBO_ERR_PARSER:
assert(0 && "Unreachable.");
}
}
static void handle_parser_error (
const GumboParserError* error,
GumboStringBuffer* output
) {
if (
error->parser_state == GUMBO_INSERTION_MODE_INITIAL
&& error->input_type != GUMBO_TOKEN_DOCTYPE
) {
print_message (
output,
"Expected a doctype token"
);
return;
}
switch (error->input_type) {
case GUMBO_TOKEN_DOCTYPE:
print_message(output, "This is not a legal doctype");
return;
case GUMBO_TOKEN_COMMENT:
// Should never happen; comments are always legal.
assert(0);
// But just in case...
print_message(output, "Comments aren't legal here");
return;
case GUMBO_TOKEN_CDATA:
case GUMBO_TOKEN_WHITESPACE:
case GUMBO_TOKEN_CHARACTER:
print_message(output, "Character tokens aren't legal here");
return;
case GUMBO_TOKEN_NULL:
print_message(output, "Null bytes are not allowed in HTML5");
return;
case GUMBO_TOKEN_EOF:
if (error->parser_state == GUMBO_INSERTION_MODE_INITIAL) {
print_message(output, "You must provide a doctype");
} else {
print_message(output, "Premature end of file.");
print_tag_stack(error, output);
}
return;
case GUMBO_TOKEN_START_TAG:
case GUMBO_TOKEN_END_TAG:
{
const char* tag_name;
const char* which = error->input_type == GUMBO_TOKEN_START_TAG ? "Start" : "End";
if (error->input_name) {
tag_name = error->input_name;
} else {
tag_name = gumbo_normalized_tagname(error->input_tag);
}
print_message(output, "%s tag '%s' isn't allowed here.", which, tag_name);
print_tag_stack(error, output);
return;
}
}
}
// Finds the preceding newline in an original source buffer from a given byte
// location. Returns a character pointer to the character after that, or a
// pointer to the beginning of the string if this is the first line.
static const char* find_prev_newline (
const char* source_text,
size_t source_length,
const char* error_location
) {
const char* source_end = source_text + source_length;
assert(error_location >= source_text);
assert(error_location <= source_end);
const char* c = error_location;
if (c != source_text && (error_location == source_end || *c == '\n'))
--c;
while (c != source_text && *c != '\n')
--c;
return c == source_text ? c : c + 1;
}
// Finds the next newline in the original source buffer from a given byte
// location. Returns a character pointer to that newline, or a pointer to
// source_text + source_length if this is the last line.
static const char* find_next_newline(
const char* source_text,
size_t source_length,
const char* error_location
) {
const char* source_end = source_text + source_length;
assert(error_location >= source_text);
assert(error_location <= source_end);
const char* c = error_location;
while (c != source_end && *c != '\n')
++c;
return c;
}
GumboError* gumbo_add_error(GumboParser* parser) {
parser->_output->document_error = true;
int max_errors = parser->_options->max_errors;
if (max_errors >= 0 && parser->_output->errors.length >= (unsigned int) max_errors) {
return NULL;
}
GumboError* error = gumbo_alloc(sizeof(GumboError));
gumbo_vector_add(error, &parser->_output->errors);
return error;
}
GumboSourcePosition gumbo_error_position(const GumboError* error) {
return error->position;
}
const char* gumbo_error_code(const GumboError* error) {
switch (error->type) {
// Defined tokenizer errors.
case GUMBO_ERR_ABRUPT_CLOSING_OF_EMPTY_COMMENT:
return "abrupt-closing-of-empty-comment";
case GUMBO_ERR_ABRUPT_DOCTYPE_PUBLIC_IDENTIFIER:
return "abrupt-doctype-public-identifier";
case GUMBO_ERR_ABRUPT_DOCTYPE_SYSTEM_IDENTIFIER:
return "abrupt-doctype-system-identifier";
case GUMBO_ERR_ABSENCE_OF_DIGITS_IN_NUMERIC_CHARACTER_REFERENCE:
return "absence-of-digits-in-numeric-character-reference";
case GUMBO_ERR_CDATA_IN_HTML_CONTENT:
return "cdata-in-html-content";
case GUMBO_ERR_CHARACTER_REFERENCE_OUTSIDE_UNICODE_RANGE:
return "character-reference-outside-unicode-range";
case GUMBO_ERR_CONTROL_CHARACTER_IN_INPUT_STREAM:
return "control-character-in-input-stream";
case GUMBO_ERR_CONTROL_CHARACTER_REFERENCE:
return "control-character-reference";
case GUMBO_ERR_END_TAG_WITH_ATTRIBUTES:
return "end-tag-with-attributes";
case GUMBO_ERR_DUPLICATE_ATTRIBUTE:
return "duplicate-attribute";
case GUMBO_ERR_END_TAG_WITH_TRAILING_SOLIDUS:
return "end-tag-with-trailing-solidus";
case GUMBO_ERR_EOF_BEFORE_TAG_NAME:
return "eof-before-tag-name";
case GUMBO_ERR_EOF_IN_CDATA:
return "eof-in-cdata";
case GUMBO_ERR_EOF_IN_COMMENT:
return "eof-in-comment";
case GUMBO_ERR_EOF_IN_DOCTYPE:
return "eof-in-doctype";
case GUMBO_ERR_EOF_IN_SCRIPT_HTML_COMMENT_LIKE_TEXT:
return "eof-in-script-html-comment-like-text";
case GUMBO_ERR_EOF_IN_TAG:
return "eof-in-tag";
case GUMBO_ERR_INCORRECTLY_CLOSED_COMMENT:
return "incorrectly-closed-comment";
case GUMBO_ERR_INCORRECTLY_OPENED_COMMENT:
return "incorrectly-opened-comment";
case GUMBO_ERR_INVALID_CHARACTER_SEQUENCE_AFTER_DOCTYPE_NAME:
return "invalid-character-sequence-after-doctype-name";
case GUMBO_ERR_INVALID_FIRST_CHARACTER_OF_TAG_NAME:
return "invalid-first-character-of-tag-name";
case GUMBO_ERR_MISSING_ATTRIBUTE_VALUE:
return "missing-attribute-value";
case GUMBO_ERR_MISSING_DOCTYPE_NAME:
return "missing-doctype-name";
case GUMBO_ERR_MISSING_DOCTYPE_PUBLIC_IDENTIFIER:
return "missing-doctype-public-identifier";
case GUMBO_ERR_MISSING_DOCTYPE_SYSTEM_IDENTIFIER:
return "missing-doctype-system-identifier";
case GUMBO_ERR_MISSING_END_TAG_NAME:
return "missing-end-tag-name";
case GUMBO_ERR_MISSING_QUOTE_BEFORE_DOCTYPE_PUBLIC_IDENTIFIER:
return "missing-quote-before-doctype-public-identifier";
case GUMBO_ERR_MISSING_QUOTE_BEFORE_DOCTYPE_SYSTEM_IDENTIFIER:
return "missing-quote-before-doctype-system-identifier";
case GUMBO_ERR_MISSING_SEMICOLON_AFTER_CHARACTER_REFERENCE:
return "missing-semicolon-after-character-reference";
case GUMBO_ERR_MISSING_WHITESPACE_AFTER_DOCTYPE_PUBLIC_KEYWORD:
return "missing-whitespace-after-doctype-public-keyword";
case GUMBO_ERR_MISSING_WHITESPACE_AFTER_DOCTYPE_SYSTEM_KEYWORD:
return "missing-whitespace-after-doctype-system-keyword";
case GUMBO_ERR_MISSING_WHITESPACE_BEFORE_DOCTYPE_NAME:
return "missing-whitespace-before-doctype-name";
case GUMBO_ERR_MISSING_WHITESPACE_BETWEEN_ATTRIBUTES:
return "missing-whitespace-between-attributes";
case GUMBO_ERR_MISSING_WHITESPACE_BETWEEN_DOCTYPE_PUBLIC_AND_SYSTEM_IDENTIFIERS:
return "missing-whitespace-between-doctype-public-and-system-identifiers";
case GUMBO_ERR_NESTED_COMMENT:
return "nested-comment";
case GUMBO_ERR_NONCHARACTER_CHARACTER_REFERENCE:
return "noncharacter-character-reference";
case GUMBO_ERR_NONCHARACTER_IN_INPUT_STREAM:
return "noncharacter-in-input-stream";
case GUMBO_ERR_NON_VOID_HTML_ELEMENT_START_TAG_WITH_TRAILING_SOLIDUS:
return "non-void-html-element-start-tag-with-trailing-solidus";
case GUMBO_ERR_NULL_CHARACTER_REFERENCE:
return "null-character-reference";
case GUMBO_ERR_SURROGATE_CHARACTER_REFERENCE:
return "surrogate-character-reference";
case GUMBO_ERR_SURROGATE_IN_INPUT_STREAM:
return "surrogate-in-input-stream";
case GUMBO_ERR_UNEXPECTED_CHARACTER_AFTER_DOCTYPE_SYSTEM_IDENTIFIER:
return "unexpected-character-after-doctype-system-identifier";
case GUMBO_ERR_UNEXPECTED_CHARACTER_IN_ATTRIBUTE_NAME:
return "unexpected-character-in-attribute-name";
case GUMBO_ERR_UNEXPECTED_CHARACTER_IN_UNQUOTED_ATTRIBUTE_VALUE:
return "unexpected-character-in-unquoted-attribute-value";
case GUMBO_ERR_UNEXPECTED_EQUALS_SIGN_BEFORE_ATTRIBUTE_NAME:
return "unexpected-equals-sign-before-attribute-name";
case GUMBO_ERR_UNEXPECTED_NULL_CHARACTER:
return "unexpected-null-character";
case GUMBO_ERR_UNEXPECTED_QUESTION_MARK_INSTEAD_OF_TAG_NAME:
return "unexpected-question-mark-instead-of-tag-name";
case GUMBO_ERR_UNEXPECTED_SOLIDUS_IN_TAG:
return "unexpected-solidus-in-tag";
case GUMBO_ERR_UNKNOWN_NAMED_CHARACTER_REFERENCE:
return "unknown-named-character-reference";
// Encoding errors.
case GUMBO_ERR_UTF8_INVALID:
return "utf8-invalid";
case GUMBO_ERR_UTF8_TRUNCATED:
return "utf8-truncated";
// Generic parser error.
case GUMBO_ERR_PARSER:
return "generic-parser";
}
// Silence warning about control reaching end of non-void function.
// All errors _should_ be handled in the switch statement.
return "generic-parser";
}
static void error_to_string (
const GumboError* error,
GumboStringBuffer* output
) {
if (error->type < GUMBO_ERR_PARSER)
handle_tokenizer_error(error, output);
else
handle_parser_error(&error->v.parser, output);
}
size_t gumbo_error_to_string(const GumboError* error, char** output) {
GumboStringBuffer sb;
gumbo_string_buffer_init(&sb);
error_to_string(error, &sb);
*output = sb.data;
return sb.length;
}
void caret_diagnostic_to_string (
const GumboError* error,
const char* source_text,
size_t source_length,
GumboStringBuffer* output
) {
error_to_string(error, output);
const char* error_text = error->original_text.data;
const char* line_start = find_prev_newline(source_text, source_length, error_text);
const char* line_end = find_next_newline(source_text, source_length, error_text);
GumboStringPiece original_line;
original_line.data = line_start;
original_line.length = line_end - line_start;
gumbo_string_buffer_append_codepoint('\n', output);
gumbo_string_buffer_append_string(&original_line, output);
gumbo_string_buffer_append_codepoint('\n', output);
gumbo_string_buffer_reserve(output->length + error->position.column, output);
if (error->position.column >= 2) {
size_t num_spaces = error->position.column - 1;
memset(output->data + output->length, ' ', num_spaces);
output->length += num_spaces;
}
gumbo_string_buffer_append_codepoint('^', output);
gumbo_string_buffer_append_codepoint('\n', output);
}
size_t gumbo_caret_diagnostic_to_string (
const GumboError* error,
const char* source_text,
size_t source_length,
char **output
) {
GumboStringBuffer sb;
gumbo_string_buffer_init(&sb);
caret_diagnostic_to_string(error, source_text, source_length, &sb);
*output = sb.data;
return sb.length;
}
void gumbo_print_caret_diagnostic (
const GumboError* error,
const char* source_text,
size_t source_length
) {
GumboStringBuffer text;
gumbo_string_buffer_init(&text);
print_message (
&text,
"%lu:%lu: ",
(unsigned long)error->position.line,
(unsigned long)error->position.column
);
caret_diagnostic_to_string(error, source_text, source_length, &text);
printf("%.*s", (int) text.length, text.data);
gumbo_string_buffer_destroy(&text);
}
void gumbo_error_destroy(GumboError* error) {
if (error->type == GUMBO_ERR_PARSER) {
// Free the tag name.
if (error->v.parser.input_name) {
gumbo_free(error->v.parser.input_name);
}
for (unsigned int i = 0; i < error->v.parser.tag_stack.length; ++i) {
intptr_t tag = (intptr_t) error->v.parser.tag_stack.data[i];
if (tag > GUMBO_TAG_UNKNOWN) {
gumbo_free(error->v.parser.tag_stack.data[i]);
}
}
gumbo_vector_destroy(&error->v.parser.tag_stack);
}
gumbo_free(error);
}
void gumbo_init_errors(GumboParser* parser) {
gumbo_vector_init(5, &parser->_output->errors);
}
void gumbo_destroy_errors(GumboParser* parser) {
for (unsigned int i = 0; i < parser->_output->errors.length; ++i) {
gumbo_error_destroy(parser->_output->errors.data[i]);
}
gumbo_vector_destroy(&parser->_output->errors);
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/error.h
Vendored
+152
View File
@@ -0,0 +1,152 @@
#ifndef GUMBO_ERROR_H_
#define GUMBO_ERROR_H_
#include <stdint.h>
#include "nokogiri_gumbo.h"
#include "insertion_mode.h"
#include "string_buffer.h"
#include "token_type.h"
#include "tokenizer_states.h"
#ifdef __cplusplus
extern "C" {
#endif
struct GumboInternalParser;
typedef enum {
// Defined errors.
// https://html.spec.whatwg.org/multipage/parsing.html#parse-errors
GUMBO_ERR_ABRUPT_CLOSING_OF_EMPTY_COMMENT,
GUMBO_ERR_ABRUPT_DOCTYPE_PUBLIC_IDENTIFIER,
GUMBO_ERR_ABRUPT_DOCTYPE_SYSTEM_IDENTIFIER,
GUMBO_ERR_ABSENCE_OF_DIGITS_IN_NUMERIC_CHARACTER_REFERENCE,
GUMBO_ERR_CDATA_IN_HTML_CONTENT,
GUMBO_ERR_CHARACTER_REFERENCE_OUTSIDE_UNICODE_RANGE,
GUMBO_ERR_CONTROL_CHARACTER_IN_INPUT_STREAM,
GUMBO_ERR_CONTROL_CHARACTER_REFERENCE,
GUMBO_ERR_END_TAG_WITH_ATTRIBUTES,
GUMBO_ERR_DUPLICATE_ATTRIBUTE,
GUMBO_ERR_END_TAG_WITH_TRAILING_SOLIDUS,
GUMBO_ERR_EOF_BEFORE_TAG_NAME,
GUMBO_ERR_EOF_IN_CDATA,
GUMBO_ERR_EOF_IN_COMMENT,
GUMBO_ERR_EOF_IN_DOCTYPE,
GUMBO_ERR_EOF_IN_SCRIPT_HTML_COMMENT_LIKE_TEXT,
GUMBO_ERR_EOF_IN_TAG,
GUMBO_ERR_INCORRECTLY_CLOSED_COMMENT,
GUMBO_ERR_INCORRECTLY_OPENED_COMMENT,
GUMBO_ERR_INVALID_CHARACTER_SEQUENCE_AFTER_DOCTYPE_NAME,
GUMBO_ERR_INVALID_FIRST_CHARACTER_OF_TAG_NAME,
GUMBO_ERR_MISSING_ATTRIBUTE_VALUE,
GUMBO_ERR_MISSING_DOCTYPE_NAME,
GUMBO_ERR_MISSING_DOCTYPE_PUBLIC_IDENTIFIER,
GUMBO_ERR_MISSING_DOCTYPE_SYSTEM_IDENTIFIER,
GUMBO_ERR_MISSING_END_TAG_NAME,
GUMBO_ERR_MISSING_QUOTE_BEFORE_DOCTYPE_PUBLIC_IDENTIFIER,
GUMBO_ERR_MISSING_QUOTE_BEFORE_DOCTYPE_SYSTEM_IDENTIFIER,
GUMBO_ERR_MISSING_SEMICOLON_AFTER_CHARACTER_REFERENCE,
GUMBO_ERR_MISSING_WHITESPACE_AFTER_DOCTYPE_PUBLIC_KEYWORD,
GUMBO_ERR_MISSING_WHITESPACE_AFTER_DOCTYPE_SYSTEM_KEYWORD,
GUMBO_ERR_MISSING_WHITESPACE_BEFORE_DOCTYPE_NAME,
GUMBO_ERR_MISSING_WHITESPACE_BETWEEN_ATTRIBUTES,
GUMBO_ERR_MISSING_WHITESPACE_BETWEEN_DOCTYPE_PUBLIC_AND_SYSTEM_IDENTIFIERS,
GUMBO_ERR_NESTED_COMMENT,
GUMBO_ERR_NONCHARACTER_CHARACTER_REFERENCE,
GUMBO_ERR_NONCHARACTER_IN_INPUT_STREAM,
GUMBO_ERR_NON_VOID_HTML_ELEMENT_START_TAG_WITH_TRAILING_SOLIDUS,
GUMBO_ERR_NULL_CHARACTER_REFERENCE,
GUMBO_ERR_SURROGATE_CHARACTER_REFERENCE,
GUMBO_ERR_SURROGATE_IN_INPUT_STREAM,
GUMBO_ERR_UNEXPECTED_CHARACTER_AFTER_DOCTYPE_SYSTEM_IDENTIFIER,
GUMBO_ERR_UNEXPECTED_CHARACTER_IN_ATTRIBUTE_NAME,
GUMBO_ERR_UNEXPECTED_CHARACTER_IN_UNQUOTED_ATTRIBUTE_VALUE,
GUMBO_ERR_UNEXPECTED_EQUALS_SIGN_BEFORE_ATTRIBUTE_NAME,
GUMBO_ERR_UNEXPECTED_NULL_CHARACTER,
GUMBO_ERR_UNEXPECTED_QUESTION_MARK_INSTEAD_OF_TAG_NAME,
GUMBO_ERR_UNEXPECTED_SOLIDUS_IN_TAG,
GUMBO_ERR_UNKNOWN_NAMED_CHARACTER_REFERENCE,
// Encoding errors.
GUMBO_ERR_UTF8_INVALID,
GUMBO_ERR_UTF8_TRUNCATED,
// Generic parser error.
GUMBO_ERR_PARSER,
} GumboErrorType;
// Additional data for tokenizer errors.
// This records the current state and codepoint encountered - this is usually
// enough to reconstruct what went wrong and provide a friendly error message.
typedef struct GumboInternalTokenizerError {
// The bad codepoint encountered.
int codepoint;
// The state that the tokenizer was in at the time.
GumboTokenizerEnum state;
} GumboTokenizerError;
// Additional data for parse errors.
typedef struct GumboInternalParserError {
// The type of input token that resulted in this error.
GumboTokenType input_type;
// The HTML tag of the input token. TAG_UNKNOWN if this was not a tag token.
GumboTag input_tag;
// The HTML tag of the input token if it was nonstandard tag token. NULL otherwise.
char *input_name;
// The insertion mode that the parser was in at the time.
GumboInsertionMode parser_state;
// The tag stack at the point of the error. Note that this is an GumboVector
// of GumboTag's *stored by value* - cast the void* to an GumboTag directly to
// get at the tag. For nonstandard tags, this is a pointer to an owned char *
// containing the tag name.
GumboVector /* GumboTag */ tag_stack;
} GumboParserError;
// The overall error struct representing an error in decoding/tokenizing/parsing
// the HTML. This contains an enumerated type flag, a source position, and then
// a union of fields containing data specific to the error.
struct GumboInternalError {
// The type of error.
GumboErrorType type;
// The position within the source file where the error occurred.
GumboSourcePosition position;
// The piece of text that caused the error.
GumboStringPiece original_text;
// Type-specific error information.
union {
// Tokenizer errors.
GumboTokenizerError tokenizer;
// Parser errors.
GumboParserError parser;
} v;
};
// Adds a new error to the parser's error list, and returns a pointer to it so
// that clients can fill out the rest of its fields. May return NULL if we're
// already over the max_errors field specified in GumboOptions.
GumboError* gumbo_add_error(struct GumboInternalParser* parser);
// Initializes the errors vector in the parser.
void gumbo_init_errors(struct GumboInternalParser* errors);
// Frees all the errors in the 'errors_' field of the parser.
void gumbo_destroy_errors(struct GumboInternalParser* errors);
// Frees the memory used for a single GumboError.
void gumbo_error_destroy(GumboError* error);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_ERROR_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/foreign_attrs.c
Vendored
+103
View File
@@ -0,0 +1,103 @@
/* ANSI-C code produced by gperf version 3.1 */
/* Command-line: gperf -m100 -n src/foreign_attrs.gperf */
/* Computed positions: -k'8-9' */
/* Filtered by: gperf-filter.sed */
#include "replacement.h"
#include "macros.h"
#include <string.h>
#define TOTAL_KEYWORDS 11
#define MIN_WORD_LENGTH 5
#define MAX_WORD_LENGTH 13
#define MIN_HASH_VALUE 0
#define MAX_HASH_VALUE 10
/* maximum key range = 11, duplicates = 0 */
static inline unsigned int
hash (register const char *str, register size_t len)
{
static const unsigned char asso_values[] =
{
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 2,
11, 1, 11, 10, 4, 4, 11, 11, 3, 11,
11, 5, 3, 11, 0, 11, 2, 11, 11, 11,
11, 2, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11
};
register unsigned int hval = 0;
switch (len)
{
default:
hval += asso_values[(unsigned char)str[8]];
/*FALLTHROUGH*/
case 8:
hval += asso_values[(unsigned char)str[7]];
/*FALLTHROUGH*/
case 7:
case 6:
case 5:
break;
}
return hval;
}
const ForeignAttrReplacement *
gumbo_get_foreign_attr_replacement (register const char *str, register size_t len)
{
static const unsigned char lengthtable[] =
{
5, 10, 13, 9, 13, 10, 11, 11, 10, 10, 8
};
static const ForeignAttrReplacement wordlist[] =
{
{"xmlns", "xmlns", GUMBO_ATTR_NAMESPACE_XMLNS},
{"xlink:href", "href", GUMBO_ATTR_NAMESPACE_XLINK},
{"xlink:arcrole", "arcrole", GUMBO_ATTR_NAMESPACE_XLINK},
{"xml:space", "space", GUMBO_ATTR_NAMESPACE_XML},
{"xlink:actuate", "actuate", GUMBO_ATTR_NAMESPACE_XLINK},
{"xlink:type", "type", GUMBO_ATTR_NAMESPACE_XLINK},
{"xlink:title", "title", GUMBO_ATTR_NAMESPACE_XLINK},
{"xmlns:xlink", "xlink", GUMBO_ATTR_NAMESPACE_XMLNS},
{"xlink:role", "role", GUMBO_ATTR_NAMESPACE_XLINK},
{"xlink:show", "show", GUMBO_ATTR_NAMESPACE_XLINK},
{"xml:lang", "lang", GUMBO_ATTR_NAMESPACE_XML}
};
if (len <= MAX_WORD_LENGTH && len >= MIN_WORD_LENGTH)
{
register unsigned int key = hash (str, len);
if (key <= MAX_HASH_VALUE)
if (len == lengthtable[key])
{
register const char *s = wordlist[key].from;
if (s && *str == *s && !memcmp (str + 1, s + 1, len - 1))
return &wordlist[key];
}
}
return 0;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/hashmap.c
Vendored
+1154
View File
File diff suppressed because it is too large Load Diff
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/hashmap.h
Vendored
+62
View File
@@ -0,0 +1,62 @@
// Copyright 2020 Joshua J Baker. All rights reserved.
// Use of this source code is governed by an MIT-style
// license that can be found in the LICENSE file.
#ifndef HASHMAP_H
#define HASHMAP_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#if defined(__cplusplus)
extern "C" {
#endif // __cplusplus
struct hashmap;
struct hashmap *hashmap_new(size_t elsize, size_t cap, uint64_t seed0,
uint64_t seed1,
uint64_t (*hash)(const void *item, uint64_t seed0, uint64_t seed1),
int (*compare)(const void *a, const void *b, void *udata),
void (*elfree)(void *item),
void *udata);
struct hashmap *hashmap_new_with_allocator(void *(*malloc)(size_t),
void *(*realloc)(void *, size_t), void (*free)(void*), size_t elsize,
size_t cap, uint64_t seed0, uint64_t seed1,
uint64_t (*hash)(const void *item, uint64_t seed0, uint64_t seed1),
int (*compare)(const void *a, const void *b, void *udata),
void (*elfree)(void *item),
void *udata);
void hashmap_free(struct hashmap *map);
void hashmap_clear(struct hashmap *map, bool update_cap);
size_t hashmap_count(struct hashmap *map);
bool hashmap_oom(struct hashmap *map);
const void *hashmap_get(struct hashmap *map, const void *item);
const void *hashmap_set(struct hashmap *map, const void *item);
const void *hashmap_delete(struct hashmap *map, const void *item);
const void *hashmap_probe(struct hashmap *map, uint64_t position);
bool hashmap_scan(struct hashmap *map, bool (*iter)(const void *item, void *udata), void *udata);
bool hashmap_iter(struct hashmap *map, size_t *i, void **item);
uint64_t hashmap_sip(const void *data, size_t len, uint64_t seed0, uint64_t seed1);
uint64_t hashmap_murmur(const void *data, size_t len, uint64_t seed0, uint64_t seed1);
uint64_t hashmap_xxhash3(const void *data, size_t len, uint64_t seed0, uint64_t seed1);
const void *hashmap_get_with_hash(struct hashmap *map, const void *key, uint64_t hash);
const void *hashmap_delete_with_hash(struct hashmap *map, const void *key, uint64_t hash);
const void *hashmap_set_with_hash(struct hashmap *map, const void *item, uint64_t hash);
void hashmap_set_grow_by_power(struct hashmap *map, size_t power);
void hashmap_set_load_factor(struct hashmap *map, double load_factor);
// DEPRECATED: use `hashmap_new_with_allocator`
void hashmap_set_allocator(void *(*malloc)(size_t), void (*free)(void*));
#if defined(__cplusplus)
}
#endif // __cplusplus
#endif // HASHMAP_H
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/insertion_mode.h
Vendored
+33
View File
@@ -0,0 +1,33 @@
#ifndef GUMBO_INSERTION_MODE_H_
#define GUMBO_INSERTION_MODE_H_
// https://html.spec.whatwg.org/multipage/parsing.html#insertion-mode
// If new enum values are added, be sure to update the kTokenHandlers
// dispatch table in parser.c.
typedef enum {
GUMBO_INSERTION_MODE_INITIAL,
GUMBO_INSERTION_MODE_BEFORE_HTML,
GUMBO_INSERTION_MODE_BEFORE_HEAD,
GUMBO_INSERTION_MODE_IN_HEAD,
GUMBO_INSERTION_MODE_IN_HEAD_NOSCRIPT,
GUMBO_INSERTION_MODE_AFTER_HEAD,
GUMBO_INSERTION_MODE_IN_BODY,
GUMBO_INSERTION_MODE_TEXT,
GUMBO_INSERTION_MODE_IN_TABLE,
GUMBO_INSERTION_MODE_IN_TABLE_TEXT,
GUMBO_INSERTION_MODE_IN_CAPTION,
GUMBO_INSERTION_MODE_IN_COLUMN_GROUP,
GUMBO_INSERTION_MODE_IN_TABLE_BODY,
GUMBO_INSERTION_MODE_IN_ROW,
GUMBO_INSERTION_MODE_IN_CELL,
GUMBO_INSERTION_MODE_IN_SELECT,
GUMBO_INSERTION_MODE_IN_SELECT_IN_TABLE,
GUMBO_INSERTION_MODE_IN_TEMPLATE,
GUMBO_INSERTION_MODE_AFTER_BODY,
GUMBO_INSERTION_MODE_IN_FRAMESET,
GUMBO_INSERTION_MODE_AFTER_FRAMESET,
GUMBO_INSERTION_MODE_AFTER_AFTER_BODY,
GUMBO_INSERTION_MODE_AFTER_AFTER_FRAMESET
} GumboInsertionMode;
#endif // GUMBO_INSERTION_MODE_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/macros.h
Vendored
+91
View File
@@ -0,0 +1,91 @@
#ifndef MACROS_H
#define MACROS_H
#if (!defined(__STDC_VERSION__) || !(__STDC_VERSION__ >= 199901L)) \
&& !defined(_WIN32) && !defined(__cplusplus)
# error C99 compiler required
#endif
#if defined(_WIN32)
# define inline __inline
# define __func__ __FUNCTION__
#endif
// Calculate the number of elements in an array.
// The extra division on the third line is a trick to help prevent
// passing a pointer to the first element of an array instead of a
// reference to the array itself.
#define ARRAY_COUNT(x) ( \
(sizeof(x) / sizeof((x)[0])) \
/ ((size_t)(!(sizeof(x) % sizeof((x)[0])))) \
)
#ifdef NDEBUG
#define UNUSED_IF_NDEBUG(x) (void)(x)
#else
#define UNUSED_IF_NDEBUG(x)
#endif
#ifdef __GNUC__
#define GNUC_AT_LEAST(major, minor) ( \
(__GNUC__ > major) \
|| ((__GNUC__ == major) && (__GNUC_MINOR__ >= minor)) )
#else
#define GNUC_AT_LEAST(major, minor) 0
#endif
#ifdef __has_attribute
#define HAS_ATTRIBUTE(x) __has_attribute(x)
#else
#define HAS_ATTRIBUTE(x) 0
#endif
#if GNUC_AT_LEAST(3, 0) || HAS_ATTRIBUTE(unused) || defined(__TINYC__)
#define UNUSED __attribute__((__unused__))
#else
#define UNUSED
#endif
#if GNUC_AT_LEAST(3, 0)
#define MALLOC __attribute__((__malloc__))
#define PRINTF(x) __attribute__((__format__(__printf__, (x), (x + 1))))
#define PURE __attribute__((__pure__))
#define CONST_FN __attribute__((__const__))
#else
#define MALLOC
#define PRINTF(x)
#define PURE
#define CONST_FN
#endif
#define UNUSED_ARG(x) unused__ ## x UNUSED
#if GNUC_AT_LEAST(3, 0) && defined(__OPTIMIZE__)
#define likely(x) __builtin_expect(!!(x), 1)
#define unlikely(x) __builtin_expect(!!(x), 0)
#else
#define likely(x) (x)
#define unlikely(x) (x)
#endif
#if GNUC_AT_LEAST(3, 3) || HAS_ATTRIBUTE(nonnull)
#define NONNULL_ARGS __attribute__((__nonnull__))
#else
#define NONNULL_ARGS
#endif
#if GNUC_AT_LEAST(3, 4) || HAS_ATTRIBUTE(warn_unused_result)
#define WARN_UNUSED_RESULT __attribute__((__warn_unused_result__))
#else
#define WARN_UNUSED_RESULT
#endif
#if GNUC_AT_LEAST(5, 0) || HAS_ATTRIBUTE(returns_nonnull)
#define RETURNS_NONNULL __attribute__((__returns_nonnull__))
#else
#define RETURNS_NONNULL
#endif
#define XMALLOC MALLOC RETURNS_NONNULL
#endif // ndef MACROS_H
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/nokogiri_gumbo.h
Vendored
+938
View File
@@ -0,0 +1,938 @@
// Copyright 2010 Google Inc.
// Copyright 2018 Craig Barnes.
// Licensed under the Apache License, version 2.0.
// We use Gumbo as a prefix for types, gumbo_ as a prefix for functions,
// GUMBO_ as a prefix for enum constants and kGumbo as a prefix for
// static constants
/**
* @file
* @mainpage Gumbo HTML Parser
*
* This provides a conformant, no-dependencies implementation of the
* [HTML5] parsing algorithm. It supports only UTF-8 -- if you need
* to parse a different encoding, run a preprocessing step to convert
* to UTF-8. It returns a parse tree made of the structs in this file.
*
* Example:
* @code
* GumboOutput* output = gumbo_parse(input);
* do_something_with_doctype(output->document);
* do_something_with_html_tree(output->root);
* gumbo_destroy_output(output);
* @endcode
*
* [HTML5]: https://html.spec.whatwg.org/multipage/
*/
#ifndef GUMBO_H
#define GUMBO_H
#include <stdbool.h>
#include <stddef.h>
#ifdef __cplusplus
extern "C" {
#endif
/**
* A struct representing a character position within the original text
* buffer. Line and column numbers are 1-based and offsets are 0-based,
* which matches how most editors and command-line tools work.
*/
typedef struct {
size_t line;
size_t column;
size_t offset;
} GumboSourcePosition;
/**
* A struct representing a string or part of a string. Strings within
* the parser are represented by a `char*` and a length; the `char*`
* points into an existing data buffer owned by some other code (often
* the original input). `GumboStringPiece`s are assumed (by convention)
* to be immutable, because they may share data. Clients should assume
* that it is not NUL-terminated and should always use explicit lengths
* when manipulating them.
*/
typedef struct {
/** A pointer to the beginning of the string. `NULL` if `length == 0`. */
const char* data;
/** The length of the string fragment, in bytes (may be zero). */
size_t length;
} GumboStringPiece;
#define GUMBO_EMPTY_STRING_INIT { .data = NULL, .length = 0 }
/** A constant to represent a 0-length null string. */
#define kGumboEmptyString (const GumboStringPiece)GUMBO_EMPTY_STRING_INIT
/**
* Compares two `GumboStringPiece`s, and returns `true` if they're
* equal or `false` otherwise.
*/
bool gumbo_string_equals (
const GumboStringPiece* str1,
const GumboStringPiece* str2
);
/**
* Compares two `GumboStringPiece`s, ignoring case, and returns `true`
* if they're equal or `false` otherwise.
*/
bool gumbo_string_equals_ignore_case (
const GumboStringPiece* str1,
const GumboStringPiece* str2
);
/**
* Check if the first `GumboStringPiece` is a prefix of the second, ignoring
* case.
*/
bool gumbo_string_prefix_ignore_case (
const GumboStringPiece* prefix,
const GumboStringPiece* str
);
/**
* A simple vector implementation. This stores a pointer to a data array
* and a length. All elements are stored as `void*`; client code must
* cast to the appropriate type. Overflows upon addition result in
* reallocation of the data array, with the size doubling to maintain
* `O(1)` amortized cost. There is no removal function, as this isn't
* needed for any of the operations within this library. Iteration can
* be done through inspecting the structure directly in a `for` loop.
*/
typedef struct {
/**
* Data elements. This points to a dynamically-allocated array of
* `capacity` elements, each a `void*` to the element itself.
*/
void** data;
/** Number of elements currently in the vector. */
unsigned int length;
/** Current array capacity. */
unsigned int capacity;
} GumboVector;
# define GUMBO_EMPTY_VECTOR_INIT { .data = NULL, .length = 0, .capacity = 0 }
/** An empty (0-length, 0-capacity) `GumboVector`. */
#define kGumboEmptyVector (const GumboVector)GUMBO_EMPTY_VECTOR_INIT
/**
* Returns the first index at which an element appears in this vector
* (testing by pointer equality), or `-1` if it never does.
*/
int gumbo_vector_index_of(GumboVector* vector, const void* element);
/**
* An `enum` for all the tags defined in the HTML5 standard. These
* correspond to the tag names themselves. Enum constants exist only
* for tags that appear in the spec itself (or for tags with special
* handling in the SVG and MathML namespaces). Any other tags appear
* as `GUMBO_TAG_UNKNOWN` and the actual tag name can be obtained
* through `original_tag`.
*
* This is mostly for API convenience, so that clients of this library
* don't need to perform a `strcasecmp` to find the normalized tag
* name. It also has efficiency benefits, by letting the parser work
* with enums instead of strings.
*/
typedef enum {
GUMBO_TAG_HTML,
GUMBO_TAG_HEAD,
GUMBO_TAG_TITLE,
GUMBO_TAG_BASE,
GUMBO_TAG_LINK,
GUMBO_TAG_META,
GUMBO_TAG_STYLE,
GUMBO_TAG_SCRIPT,
GUMBO_TAG_NOSCRIPT,
GUMBO_TAG_TEMPLATE,
GUMBO_TAG_BODY,
GUMBO_TAG_ARTICLE,
GUMBO_TAG_SECTION,
GUMBO_TAG_NAV,
GUMBO_TAG_ASIDE,
GUMBO_TAG_H1,
GUMBO_TAG_H2,
GUMBO_TAG_H3,
GUMBO_TAG_H4,
GUMBO_TAG_H5,
GUMBO_TAG_H6,
GUMBO_TAG_HGROUP,
GUMBO_TAG_HEADER,
GUMBO_TAG_FOOTER,
GUMBO_TAG_ADDRESS,
GUMBO_TAG_P,
GUMBO_TAG_HR,
GUMBO_TAG_PRE,
GUMBO_TAG_BLOCKQUOTE,
GUMBO_TAG_OL,
GUMBO_TAG_UL,
GUMBO_TAG_LI,
GUMBO_TAG_DL,
GUMBO_TAG_DT,
GUMBO_TAG_DD,
GUMBO_TAG_FIGURE,
GUMBO_TAG_FIGCAPTION,
GUMBO_TAG_MAIN,
GUMBO_TAG_DIV,
GUMBO_TAG_A,
GUMBO_TAG_EM,
GUMBO_TAG_STRONG,
GUMBO_TAG_SMALL,
GUMBO_TAG_S,
GUMBO_TAG_CITE,
GUMBO_TAG_Q,
GUMBO_TAG_DFN,
GUMBO_TAG_ABBR,
GUMBO_TAG_DATA,
GUMBO_TAG_TIME,
GUMBO_TAG_CODE,
GUMBO_TAG_VAR,
GUMBO_TAG_SAMP,
GUMBO_TAG_KBD,
GUMBO_TAG_SUB,
GUMBO_TAG_SUP,
GUMBO_TAG_I,
GUMBO_TAG_B,
GUMBO_TAG_U,
GUMBO_TAG_MARK,
GUMBO_TAG_RUBY,
GUMBO_TAG_RT,
GUMBO_TAG_RP,
GUMBO_TAG_BDI,
GUMBO_TAG_BDO,
GUMBO_TAG_SPAN,
GUMBO_TAG_BR,
GUMBO_TAG_WBR,
GUMBO_TAG_INS,
GUMBO_TAG_DEL,
GUMBO_TAG_IMAGE,
GUMBO_TAG_IMG,
GUMBO_TAG_IFRAME,
GUMBO_TAG_EMBED,
GUMBO_TAG_OBJECT,
GUMBO_TAG_PARAM,
GUMBO_TAG_VIDEO,
GUMBO_TAG_AUDIO,
GUMBO_TAG_SOURCE,
GUMBO_TAG_TRACK,
GUMBO_TAG_CANVAS,
GUMBO_TAG_MAP,
GUMBO_TAG_AREA,
GUMBO_TAG_MATH,
GUMBO_TAG_MI,
GUMBO_TAG_MO,
GUMBO_TAG_MN,
GUMBO_TAG_MS,
GUMBO_TAG_MTEXT,
GUMBO_TAG_MGLYPH,
GUMBO_TAG_MALIGNMARK,
GUMBO_TAG_ANNOTATION_XML,
GUMBO_TAG_SVG,
GUMBO_TAG_FOREIGNOBJECT,
GUMBO_TAG_DESC,
GUMBO_TAG_TABLE,
GUMBO_TAG_CAPTION,
GUMBO_TAG_COLGROUP,
GUMBO_TAG_COL,
GUMBO_TAG_TBODY,
GUMBO_TAG_THEAD,
GUMBO_TAG_TFOOT,
GUMBO_TAG_TR,
GUMBO_TAG_TD,
GUMBO_TAG_TH,
GUMBO_TAG_FORM,
GUMBO_TAG_FIELDSET,
GUMBO_TAG_LEGEND,
GUMBO_TAG_LABEL,
GUMBO_TAG_INPUT,
GUMBO_TAG_BUTTON,
GUMBO_TAG_SELECT,
GUMBO_TAG_DATALIST,
GUMBO_TAG_OPTGROUP,
GUMBO_TAG_OPTION,
GUMBO_TAG_TEXTAREA,
GUMBO_TAG_KEYGEN,
GUMBO_TAG_OUTPUT,
GUMBO_TAG_PROGRESS,
GUMBO_TAG_METER,
GUMBO_TAG_DETAILS,
GUMBO_TAG_SUMMARY,
GUMBO_TAG_MENU,
GUMBO_TAG_MENUITEM,
GUMBO_TAG_APPLET,
GUMBO_TAG_ACRONYM,
GUMBO_TAG_BGSOUND,
GUMBO_TAG_DIR,
GUMBO_TAG_FRAME,
GUMBO_TAG_FRAMESET,
GUMBO_TAG_NOFRAMES,
GUMBO_TAG_LISTING,
GUMBO_TAG_XMP,
GUMBO_TAG_NEXTID,
GUMBO_TAG_NOEMBED,
GUMBO_TAG_PLAINTEXT,
GUMBO_TAG_RB,
GUMBO_TAG_STRIKE,
GUMBO_TAG_BASEFONT,
GUMBO_TAG_BIG,
GUMBO_TAG_BLINK,
GUMBO_TAG_CENTER,
GUMBO_TAG_FONT,
GUMBO_TAG_MARQUEE,
GUMBO_TAG_MULTICOL,
GUMBO_TAG_NOBR,
GUMBO_TAG_SPACER,
GUMBO_TAG_TT,
GUMBO_TAG_RTC,
GUMBO_TAG_DIALOG,
GUMBO_TAG_SEARCH,
// Used for all tags that don't have special handling in HTML.
GUMBO_TAG_UNKNOWN,
// A marker value to indicate the end of the enum, for iterating over it.
GUMBO_TAG_LAST,
} GumboTag;
/**
* Returns the normalized (all lower case) tag name for a `GumboTag` enum. The
* return value is static data owned by the library.
*/
const char* gumbo_normalized_tagname(GumboTag tag);
/**
* Extracts the tag name from the `original_text` field of an element
* or token by stripping off `</>` characters and attributes and
* adjusting the passed-in `GumboStringPiece` appropriately. The tag
* name is in the original case and shares a buffer with the original
* text, to simplify memory management. Behavior is undefined if a
* string piece that doesn't represent an HTML tag (`<tagname>` or
* `</tagname>`) is passed in. If the string piece is completely
* empty (`NULL` data pointer), then this function will exit
* successfully as a no-op.
*/
void gumbo_tag_from_original_text(GumboStringPiece* text);
/**
* Converts a tag name string (which may be in upper or mixed case) to a
* tag enum.
*/
GumboTag gumbo_tagn_enum(const char* tagname, size_t length);
/**
* Attribute namespaces.
* HTML includes special handling for XLink, XML, and XMLNS namespaces
* on attributes. Everything else goes in the generic "NONE" namespace.
*/
typedef enum {
GUMBO_ATTR_NAMESPACE_NONE,
GUMBO_ATTR_NAMESPACE_XLINK,
GUMBO_ATTR_NAMESPACE_XML,
GUMBO_ATTR_NAMESPACE_XMLNS,
} GumboAttributeNamespaceEnum;
/**
* A struct representing a single attribute on a HTML tag. This is a
* name-value pair, but also includes information about source locations
* and original source text.
*/
typedef struct {
/**
* The namespace for the attribute. This will usually be
* `GUMBO_ATTR_NAMESPACE_NONE`, but some XLink/XMLNS/XML attributes
* take special values, per:
* https://html.spec.whatwg.org/multipage/parsing.html#adjust-foreign-attributes
*/
GumboAttributeNamespaceEnum attr_namespace;
/**
* The name of the attribute. This is in a freshly-allocated buffer to
* deal with case-normalization and is null-terminated.
*/
const char* name;
/**
* The original text of the attribute name, as a pointer into the
* original source buffer.
*/
GumboStringPiece original_name;
/**
* The value of the attribute. This is in a freshly-allocated buffer
* to deal with unescaping and is null-terminated. It does not include
* any quotes that surround the attribute. If the attribute has no
* value (for example, `selected` on a checkbox) this will be an empty
* string.
*/
const char* value;
/**
* The original text of the value of the attribute. This points into
* the original source buffer. It includes any quotes that surround
* the attribute and you can look at `original_value.data[0]` and
* `original_value.data[original_value.length - 1]` to determine what
* the quote characters were. If the attribute has no value this will
* be a 0-length string.
*/
GumboStringPiece original_value;
/** The starting position of the attribute name. */
GumboSourcePosition name_start;
/**
* The ending position of the attribute name. This is not always derivable
* from the starting position of the value because of the possibility of
* whitespace around the `=` sign.
*/
GumboSourcePosition name_end;
/** The starting position of the attribute value. */
GumboSourcePosition value_start;
/** The ending position of the attribute value. */
GumboSourcePosition value_end;
} GumboAttribute;
/**
* Given a vector of `GumboAttribute`s, look up the one with the
* specified name and return it, or `NULL` if no such attribute exists.
* This uses a case-insensitive match, as HTML is case-insensitive.
*/
GumboAttribute* gumbo_get_attribute(const GumboVector* attrs, const char* name);
/**
* Enum denoting the type of node. This determines the type of the
* `node.v` union.
*/
typedef enum {
/** Document node. `v` will be a `GumboDocument`. */
GUMBO_NODE_DOCUMENT,
/** Element node. `v` will be a `GumboElement`. */
GUMBO_NODE_ELEMENT,
/** Text node. `v` will be a `GumboText`. */
GUMBO_NODE_TEXT,
/** CDATA node. `v` will be a `GumboText`. */
GUMBO_NODE_CDATA,
/** Comment node. `v` will be a `GumboText`, excluding comment delimiters. */
GUMBO_NODE_COMMENT,
/** Text node, where all contents is whitespace. `v` will be a `GumboText`. */
GUMBO_NODE_WHITESPACE,
/**
* Template node. This is separate from `GUMBO_NODE_ELEMENT` because
* many client libraries will want to ignore the contents of template
* nodes, as the spec suggests. Recursing on `GUMBO_NODE_ELEMENT` will
* do the right thing here, while clients that want to include template
* contents should also check for `GUMBO_NODE_TEMPLATE`. `v` will be a
* `GumboElement`.
*/
GUMBO_NODE_TEMPLATE
} GumboNodeType;
/**
* Forward declaration of GumboNode so it can be used recursively in
* GumboNode.parent.
*/
typedef struct GumboInternalNode GumboNode;
/** https://dom.spec.whatwg.org/#concept-document-quirks */
typedef enum {
GUMBO_DOCTYPE_NO_QUIRKS,
GUMBO_DOCTYPE_QUIRKS,
GUMBO_DOCTYPE_LIMITED_QUIRKS
} GumboQuirksModeEnum;
/**
* Namespaces.
* Unlike in X(HT)ML, namespaces in HTML5 are not denoted by a prefix.
* Rather, anything inside an `<svg>` tag is in the SVG namespace,
* anything inside the `<math>` tag is in the MathML namespace, and
* anything else is inside the HTML namespace. No other namespaces are
* supported, so this can be an `enum`.
*/
typedef enum {
GUMBO_NAMESPACE_HTML,
GUMBO_NAMESPACE_SVG,
GUMBO_NAMESPACE_MATHML
} GumboNamespaceEnum;
/**
* Parse flags.
* We track the reasons for parser insertion of nodes and store them in
* a bitvector in the node itself. This lets client code optimize out
* nodes that are implied by the HTML structure of the document, or flag
* constructs that may not be allowed by a style guide, or track the
* prevalence of incorrect or tricky HTML code.
*/
typedef enum {
/**
* A normal node -- both start and end tags appear in the source,
* nothing has been reparented.
*/
GUMBO_INSERTION_NORMAL = 0,
/**
* A node inserted by the parser to fulfill some implicit insertion
* rule. This is usually set in addition to some other flag giving a
* more specific insertion reason; it's a generic catch-all term
* meaning "The start tag for this node did not appear in the document
* source".
*/
GUMBO_INSERTION_BY_PARSER = 1 << 0,
/**
* A flag indicating that the end tag for this node did not appear in
* the document source. Note that in some cases, you can still have
* parser-inserted nodes with an explicit end tag. For example,
* `Text</html>` has `GUMBO_INSERTED_BY_PARSER` set on the `<html>`
* node, but `GUMBO_INSERTED_END_TAG_IMPLICITLY` is unset, as the
* `</html>` tag actually exists.
*
* This flag will be set only if the end tag is completely missing.
* In some cases, the end tag may be misplaced (e.g. a `</body>` tag
* with text afterwards), which will leave this flag unset and require
* clients to inspect the parse errors for that case.
*/
GUMBO_INSERTION_IMPLICIT_END_TAG = 1 << 1,
// Value 1 << 2 was for a flag that has since been removed.
/**
* A flag for nodes that are inserted because their presence is
* implied by other tags, e.g. `<html>`, `<head>`, `<body>`,
* `<tbody>`, etc.
*/
GUMBO_INSERTION_IMPLIED = 1 << 3,
/**
* A flag for nodes that are converted from their end tag equivalents.
* For example, `</p>` when no paragraph is open implies that the
* parser should create a `<p>` tag and immediately close it, while
* `</br>` means the same thing as `<br>`.
*/
GUMBO_INSERTION_CONVERTED_FROM_END_TAG = 1 << 4,
// Value 1 << 5 was for a flag that has since been removed.
/** A flag for `<image>` tags that are rewritten as `<img>`. */
GUMBO_INSERTION_FROM_IMAGE = 1 << 6,
/**
* A flag for nodes that are cloned as a result of the reconstruction
* of active formatting elements. This is set only on the clone; the
* initial portion of the formatting run is a NORMAL node with an
* `IMPLICIT_END_TAG`.
*/
GUMBO_INSERTION_RECONSTRUCTED_FORMATTING_ELEMENT = 1 << 7,
/** A flag for nodes that are cloned by the adoption agency algorithm. */
GUMBO_INSERTION_ADOPTION_AGENCY_CLONED = 1 << 8,
/** A flag for nodes that are moved by the adoption agency algorithm. */
GUMBO_INSERTION_ADOPTION_AGENCY_MOVED = 1 << 9,
/**
* A flag for nodes that have been foster-parented out of a table (or
* should've been foster-parented, if verbatim mode is set).
*/
GUMBO_INSERTION_FOSTER_PARENTED = 1 << 10,
} GumboParseFlags;
/** Information specific to document nodes. */
typedef struct {
/**
* An array of `GumboNode`s, containing the children of this element.
* This will normally consist of the `<html>` element and any comment
* nodes found. Pointers are owned.
*/
GumboVector /* GumboNode* */ children;
/**
* `true` if there was an explicit doctype token, as opposed to it
* being omitted.
*/
bool has_doctype;
// Fields from the doctype token, copied verbatim.
const char* name;
const char* public_identifier;
const char* system_identifier;
/**
* Whether or not the document is in QuirksMode, as determined by the
* values in the GumboTokenDocType template.
*/
GumboQuirksModeEnum doc_type_quirks_mode;
} GumboDocument;
/**
* The struct used to represent TEXT, CDATA, COMMENT, and WHITESPACE
* elements. This contains just a block of text and its position.
*/
typedef struct {
/**
* The text of this node, after entities have been parsed and decoded.
* For comment and cdata nodes, this does not include the comment
* delimiters.
*/
const char* text;
/**
* The original text of this node, as a pointer into the original
* buffer. For comment/cdata nodes, this includes the comment
* delimiters.
*/
GumboStringPiece original_text;
/**
* The starting position of this node. This corresponds to the
* position of `original_text`, before entities are decoded.
* */
GumboSourcePosition start_pos;
} GumboText;
/**
* The struct used to represent all HTML elements. This contains
* information about the tag, attributes, and child nodes.
*/
typedef struct {
/**
* An array of `GumboNode`s, containing the children of this element.
* Pointers are owned.
*/
GumboVector /* GumboNode* */ children;
/** The GumboTag enum for this element. */
GumboTag tag;
/** The name for this element. */
const char* name;
/** The GumboNamespaceEnum for this element. */
GumboNamespaceEnum tag_namespace;
/**
* A `GumboStringPiece` pointing to the original tag text for this
* element, pointing directly into the source buffer. If the tag was
* inserted algorithmically (for example, `<head>` or `<tbody>`
* insertion), this will be a zero-length string.
*/
GumboStringPiece original_tag;
/**
* A `GumboStringPiece` pointing to the original end tag text for this
* element. If the end tag was inserted algorithmically, (for example,
* closing a self-closing tag), this will be a zero-length string.
*/
GumboStringPiece original_end_tag;
/** The source position for the start of the start tag. */
GumboSourcePosition start_pos;
/** The source position for the start of the end tag. */
GumboSourcePosition end_pos;
/**
* An array of `GumboAttribute`s, containing the attributes for this
* tag in the order that they were parsed. Pointers are owned.
*/
GumboVector /* GumboAttribute* */ attributes;
} GumboElement;
/**
* A supertype for `GumboElement` and `GumboText`, so that we can
* include one generic type in lists of children and cast as necessary
* to subtypes.
*/
struct GumboInternalNode {
/** The type of node that this is. */
GumboNodeType type;
/** Pointer back to parent node. Not owned. */
GumboNode* parent;
/** The index within the parent's children vector of this node. */
unsigned int index_within_parent;
/**
* A bitvector of flags containing information about why this element
* was inserted into the parse tree, including a variety of special
* parse situations.
*/
GumboParseFlags parse_flags;
/** The actual node data. */
union {
GumboDocument document; // For GUMBO_NODE_DOCUMENT.
GumboElement element; // For GUMBO_NODE_ELEMENT.
GumboText text; // For everything else.
} v;
};
/**
* Input struct containing configuration options for the parser.
* These let you specify alternate memory managers, provide different
* error handling, etc. Use `kGumboDefaultOptions` for sensible
* defaults and only set what you need.
*/
typedef struct GumboInternalOptions {
/**
* The tab-stop size, for computing positions in HTML files that
* use tabs. Default: `8`.
*/
int tab_stop;
/**
* Whether or not to stop parsing when the first error is encountered.
* Default: `false`.
*/
bool stop_on_first_error;
/**
* Maximum allowed number of attributes per element. If this limit is
* exceeded, the parser will return early with a partial document and
* the returned `GumboOutput` will have its `status` field set to
* `GUMBO_STATUS_TOO_MANY_ATTRIBUTES`. Set to `-1` to disable the limit.
* Default: `400`.
*/
int max_attributes;
/**
* Maximum allowed depth for the parse tree. If this limit is exceeded,
* the parser will return early with a partial document and the returned
* `GumboOutput` will have its `status` field set to
* `GUMBO_STATUS_TREE_TOO_DEEP`.
* Default: `400`.
*/
unsigned int max_tree_depth;
/**
* The maximum number of errors before the parser stops recording
* them. This is provided so that if the page is totally borked, we
* don't completely fill up the errors vector and exhaust memory with
* useless redundant errors. Set to `-1` to disable the limit.
* Default: `-1`.
*/
int max_errors;
/**
* The fragment context for parsing:
* https://html.spec.whatwg.org/multipage/parsing.html#parsing-html-fragments
*
* If `NULL` is passed here, it is assumed to be "no
* fragment", i.e. the regular parsing algorithm. Otherwise, pass the
* tag name for the intended parent of the parsed fragment. We use the
* tag name, namespace, and encoding attribute which are sufficient to
* set all of the parsing context needed for fragment parsing.
*
* Default: `NULL`.
*/
const char* fragment_context;
/**
* The namespace for the fragment context. This lets client code
* differentiate between, say, parsing a `<title>` tag in SVG vs.
* parsing it in HTML.
*
* Default: `GUMBO_NAMESPACE_HTML`.
*/
GumboNamespaceEnum fragment_namespace;
/**
* The value of the fragment context's `encoding` attribute, if any.
* Set to `NULL` for no `encoding` attribute.
*
* Default: `NULL`.
*/
const char* fragment_encoding;
/**
* Quirks mode for fragment parsing. The quirks mode for a given DOCTYPE can
* be looked up using `gumbo_compute_quirks_mode()`.
*
* Default: `GUMBO_DOCTYPE_NO_QUIRKS`.
*/
GumboQuirksModeEnum quirks_mode;
/**
* For fragment parsing. Set this to true if the context node has a form
* element as an ancestor.
*
* Default: `false`.
*/
bool fragment_context_has_form_ancestor;
/**
* Parse `noscript` elements as if scripting was enabled. This causes the
* contents of the `noscript` element to be parsed as raw text, rather
* than as HTML elements.
*
* Default: `false`.
*/
bool parse_noscript_content_as_text;
} GumboOptions;
/** Default options struct; use this with gumbo_parse_with_options. */
extern const GumboOptions kGumboDefaultOptions;
/**
* Status code indicating whether parsing finished successfully or
* was stopped mid-document due to exceptional circumstances.
*/
typedef enum {
/**
* Indicates that parsing completed successfully. The resulting tree
* will be a complete document.
*/
GUMBO_STATUS_OK,
/**
* Indicates that the maximum element nesting limit
* (`GumboOptions::max_tree_depth`) was reached during parsing. The
* resulting tree will be a partial document, with no further nodes
* created after the point where the limit was reached. The partial
* document may be useful for constructing an error message but
* typically shouldn't be used for other purposes.
*/
GUMBO_STATUS_TREE_TOO_DEEP,
/**
* Indicates that the maximum number of attributes per element
* (`GumboOptions::max_attributes`) was reached during parsing. The
* resulting tree will be a partial document, with no further nodes
* created after the point where the limit was reached. The partial
* document may be useful for constructing an error message but
* typically shouldn't be used for other purposes.
*/
GUMBO_STATUS_TOO_MANY_ATTRIBUTES,
// Currently unused
GUMBO_STATUS_OUT_OF_MEMORY,
} GumboOutputStatus;
/** The output struct containing the results of the parse. */
typedef struct GumboInternalOutput {
/**
* Pointer to the document node. This is a `GumboNode` of type
* `NODE_DOCUMENT` that contains the entire document as its child.
*/
GumboNode* document;
/**
* Pointer to the root node. This is the `<html>` tag that forms the
* root of the document.
*/
GumboNode* root;
/**
* A list of errors that occurred during the parse.
*/
GumboVector /* GumboError */ errors;
/**
* True if the parser encountered an error.
*
* This can be true and `errors` an empty `GumboVector` if the `max_errors`
* option was set to 0.
*/
bool document_error;
/**
* A status code indicating whether parsing finished successfully or was
* stopped mid-document due to exceptional circumstances.
*/
GumboOutputStatus status;
} GumboOutput;
/**
* Parses a buffer of UTF-8 text into an `GumboNode` parse tree. The
* buffer must live at least as long as the parse tree, as some fields
* (eg. `original_text`) point directly into the original buffer.
*
* This doesn't support buffers longer than 4 gigabytes.
*/
GumboOutput* gumbo_parse(const char* buffer);
/**
* Extended version of `gumbo_parse` that takes an explicit options
* structure, buffer, and length.
*/
GumboOutput* gumbo_parse_with_options (
const GumboOptions* options,
const char* buffer,
size_t buffer_length
);
/**
* Compute the quirks mode based on the name, public identifier, and system
* identifier. Any of these may be `NULL` to indicate a missing value.
*/
GumboQuirksModeEnum gumbo_compute_quirks_mode (
const char *name,
const char *pubid,
const char *sysid
);
/** Convert a `GumboOutputStatus` code into a readable description. */
const char* gumbo_status_to_string(GumboOutputStatus status);
/** Release the memory used for the parse tree and parse errors. */
void gumbo_destroy_output(GumboOutput* output);
/** Opaque GumboError type */
typedef struct GumboInternalError GumboError;
/**
* Returns the position of the error.
*/
GumboSourcePosition gumbo_error_position(const GumboError* error);
/**
* Returns a constant string representation of the error's code. This is owned
* by the library and should not be freed by the caller.
*/
const char* gumbo_error_code(const GumboError* error);
/**
* Prints an error to a string. This stores a freshly-allocated buffer
* containing the error message text in output. The caller is responsible for
* freeing the buffer. The size of the error message is returned. The error
* message itself may not be NULL-terminated and may contain NULL bytes so the
* returned size must be used.
*/
size_t gumbo_error_to_string(const GumboError* error, char **output);
/**
* Prints a caret diagnostic to a string. This stores a freshly-allocated
* buffer containing the error message text in output. The caller is responsible for
* freeing the buffer. The size of the error message is returned. The error
* message itself may not be NULL-terminated and may contain NULL bytes so the
* returned size must be used.
*/
size_t gumbo_caret_diagnostic_to_string (
const GumboError* error,
const char* source_text,
size_t source_length,
char** output
);
/**
* Like gumbo_caret_diagnostic_to_string, but prints the text to stdout
* instead of writing to a string.
*/
void gumbo_print_caret_diagnostic (
const GumboError* error,
const char* source_text,
size_t source_length
);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_H
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/parser.c
Vendored
+4924
View File
File diff suppressed because it is too large Load Diff
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/parser.h
Vendored
+41
View File
@@ -0,0 +1,41 @@
#ifndef GUMBO_PARSER_H_
#define GUMBO_PARSER_H_
#ifdef __cplusplus
extern "C" {
#endif
// Contains the definition of the top-level GumboParser structure that's
// threaded through basically every internal function in the library.
struct GumboInternalParserState;
struct GumboInternalOutput;
struct GumboInternalOptions;
struct GumboInternalTokenizerState;
// An overarching struct that's threaded through (nearly) all functions in the
// library, OOP-style. This gives each function access to the options and
// output, along with any internal state needed for the parse.
typedef struct GumboInternalParser {
// Settings for this parse run.
const struct GumboInternalOptions* _options;
// Output for the parse.
struct GumboInternalOutput* _output;
// The internal tokenizer state, defined as a pointer to avoid a cyclic
// dependency on html5tokenizer.h. The main parse routine is responsible for
// initializing this on parse start, and destroying it on parse end.
// End-users will never see a non-garbage value in this pointer.
struct GumboInternalTokenizerState* _tokenizer_state;
// The internal parser state. Initialized on parse start and destroyed on
// parse end; end-users will never see a non-garbage value in this pointer.
struct GumboInternalParserState* _parser_state;
} GumboParser;
#ifdef __cplusplus
}
#endif
#endif // GUMBO_PARSER_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/replacement.h
Vendored
+33
View File
@@ -0,0 +1,33 @@
#ifndef GUMBO_REPLACEMENT_H_
#define GUMBO_REPLACEMENT_H_
#include <stddef.h>
#include "nokogiri_gumbo.h"
typedef struct {
const char *const from;
const char *const to;
} StringReplacement;
const StringReplacement *gumbo_get_svg_tag_replacement (
const char* str,
size_t len
);
const StringReplacement *gumbo_get_svg_attr_replacement (
const char* str,
size_t len
);
typedef struct {
const char *const from;
const char *const local_name;
const GumboAttributeNamespaceEnum attr_namespace;
} ForeignAttrReplacement;
const ForeignAttrReplacement *gumbo_get_foreign_attr_replacement (
const char* str,
size_t len
);
#endif // GUMBO_REPLACEMENT_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/string_buffer.c
Vendored
+103
View File
@@ -0,0 +1,103 @@
/*
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <string.h>
#include "string_buffer.h"
#include "util.h"
// Size chosen via statistical analysis of ~60K websites.
// 99% of text nodes and 98% of attribute names/values fit in this initial size.
static const size_t kDefaultStringBufferSize = 5;
static void maybe_resize_string_buffer (
size_t additional_chars,
GumboStringBuffer* buffer
) {
size_t new_length = buffer->length + additional_chars;
size_t new_capacity = buffer->capacity;
while (new_capacity < new_length) {
new_capacity *= 2;
}
if (new_capacity != buffer->capacity) {
buffer->data = gumbo_realloc(buffer->data, new_capacity);
buffer->capacity = new_capacity;
}
}
void gumbo_string_buffer_init(GumboStringBuffer* output) {
output->data = gumbo_alloc(kDefaultStringBufferSize);
output->length = 0;
output->capacity = kDefaultStringBufferSize;
}
void gumbo_string_buffer_reserve (
size_t min_capacity,
GumboStringBuffer* output
) {
maybe_resize_string_buffer(min_capacity - output->length, output);
}
void gumbo_string_buffer_append_codepoint (
int c,
GumboStringBuffer* output
) {
// num_bytes is actually the number of continuation bytes, 1 less than the
// total number of bytes. This is done to keep the loop below simple and
// should probably change if we unroll it.
int num_bytes, prefix;
if (c <= 0x7f) {
num_bytes = 0;
prefix = 0;
} else if (c <= 0x7ff) {
num_bytes = 1;
prefix = 0xc0;
} else if (c <= 0xffff) {
num_bytes = 2;
prefix = 0xe0;
} else {
num_bytes = 3;
prefix = 0xf0;
}
maybe_resize_string_buffer(num_bytes + 1, output);
output->data[output->length++] = prefix | (c >> (num_bytes * 6));
for (int i = num_bytes - 1; i >= 0; --i) {
output->data[output->length++] = 0x80 | (0x3f & (c >> (i * 6)));
}
}
void gumbo_string_buffer_append_string (
const GumboStringPiece* str,
GumboStringBuffer* output
) {
maybe_resize_string_buffer(str->length, output);
memcpy(output->data + output->length, str->data, str->length);
output->length += str->length;
}
char* gumbo_string_buffer_to_string(const GumboStringBuffer* input) {
char* buffer = gumbo_alloc(input->length + 1);
memcpy(buffer, input->data, input->length);
buffer[input->length] = '\0';
return buffer;
}
void gumbo_string_buffer_clear(GumboStringBuffer* input) {
input->length = 0;
}
void gumbo_string_buffer_destroy(GumboStringBuffer* buffer) {
gumbo_free(buffer->data);
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/string_buffer.h
Vendored
+68
View File
@@ -0,0 +1,68 @@
#ifndef GUMBO_STRING_BUFFER_H_
#define GUMBO_STRING_BUFFER_H_
#include <stdbool.h>
#include <stddef.h>
#include "nokogiri_gumbo.h"
#ifdef __cplusplus
extern "C" {
#endif
// A struct representing a mutable, growable string. This consists of a
// heap-allocated buffer that may grow (by doubling) as necessary. When
// converting to a string, this allocates a new buffer that is only as long as
// it needs to be. Note that the internal buffer here is *not* nul-terminated,
// so be sure not to use ordinary string manipulation functions on it.
typedef struct {
// A pointer to the beginning of the string. NULL if length == 0.
char* data;
// The length of the string fragment, in bytes. May be zero.
size_t length;
// The capacity of the buffer, in bytes.
size_t capacity;
} GumboStringBuffer;
// Initializes a new GumboStringBuffer.
void gumbo_string_buffer_init(GumboStringBuffer* output);
// Ensures that the buffer contains at least a certain amount of space. Most
// useful with snprintf and the other length-delimited string functions, which
// may want to write directly into the buffer.
void gumbo_string_buffer_reserve (
size_t min_capacity,
GumboStringBuffer* output
);
// Appends a single Unicode codepoint onto the end of the GumboStringBuffer.
// This is essentially a UTF-8 encoder, and may add 1-4 bytes depending on the
// value of the codepoint.
void gumbo_string_buffer_append_codepoint (
int c,
GumboStringBuffer* output
);
// Appends a string onto the end of the GumboStringBuffer.
void gumbo_string_buffer_append_string (
const GumboStringPiece* str,
GumboStringBuffer* output
);
// Converts this string buffer to const char*, alloctaing a new buffer for it.
char* gumbo_string_buffer_to_string(const GumboStringBuffer* input);
// Reinitialize this string buffer. This clears it by setting length=0. It
// does not zero out the buffer itself.
void gumbo_string_buffer_clear(GumboStringBuffer* input);
// Deallocates this GumboStringBuffer.
void gumbo_string_buffer_destroy(GumboStringBuffer* buffer);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_STRING_BUFFER_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/string_piece.c
Vendored
+48
View File
@@ -0,0 +1,48 @@
/*
Copyright 2018 Craig Barnes.
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <stddef.h>
#include <string.h>
#include "nokogiri_gumbo.h"
#include "ascii.h"
bool gumbo_string_equals (
const GumboStringPiece* str1,
const GumboStringPiece* str2
) {
return
str1->length == str2->length
&& !memcmp(str1->data, str2->data, str1->length);
}
bool gumbo_string_equals_ignore_case (
const GumboStringPiece* str1,
const GumboStringPiece* str2
) {
return
str1->length == str2->length
&& !gumbo_ascii_strncasecmp(str1->data, str2->data, str1->length);
}
bool gumbo_string_prefix_ignore_case (
const GumboStringPiece* prefix,
const GumboStringPiece* str
) {
return
prefix->length <= str->length
&& !gumbo_ascii_strncasecmp(prefix->data, str->data, prefix->length);
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/string_set.c
Vendored
+41
View File
@@ -0,0 +1,41 @@
#include "string_set.h"
#include <string.h>
#include "hashmap.h"
#define SEED0 0xf00ba2
#define SEED1 0xfa1afe1
static int
string_compare(const void *a, const void *b, void *udata) {
return strcmp(*(const char **)a, *(const char **)b);
}
static uint64_t
string_hash(const void *item, uint64_t seed0, uint64_t seed1) {
const char *str = *(const char **)item;
return hashmap_xxhash3(str, strlen(str), seed0, seed1);
}
GumboStringSet *
gumbo_string_set_new(size_t cap)
{
return hashmap_new(sizeof(char *), cap, SEED0, SEED1, string_hash, string_compare, NULL, NULL);
}
void gumbo_string_set_free(GumboStringSet *set)
{
hashmap_free(set);
}
void
gumbo_string_set_insert(GumboStringSet *set, const char *str)
{
hashmap_set(set, &str);
}
int
gumbo_string_set_contains(GumboStringSet *set, const char *str)
{
return hashmap_get(set, &str) == NULL ? 0 : 1;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/string_set.h
Vendored
+21
View File
@@ -0,0 +1,21 @@
#ifndef STRING_SET_H
#define STRING_SET_H
#include <stddef.h>
#if defined(__cplusplus)
extern "C" {
#endif // __cplusplus
typedef struct hashmap GumboStringSet;
GumboStringSet* gumbo_string_set_new(size_t cap);
void gumbo_string_set_free(GumboStringSet *set);
void gumbo_string_set_insert(GumboStringSet *set, const char *str);
int gumbo_string_set_contains(GumboStringSet *set, const char *str);
#if defined(__cplusplus)
}
#endif // __cplusplus
#endif // STRING_SET_H
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/svg_attrs.c
Vendored
+174
View File
@@ -0,0 +1,174 @@
/* ANSI-C code produced by gperf version 3.1 */
/* Command-line: gperf -m100 src/svg_attrs.gperf */
/* Computed positions: -k'1,10,$' */
/* Filtered by: gperf-filter.sed */
#include "replacement.h"
#include "macros.h"
#include "ascii.h"
#include <string.h>
#define TOTAL_KEYWORDS 58
#define MIN_WORD_LENGTH 4
#define MAX_WORD_LENGTH 19
#define MIN_HASH_VALUE 5
#define MAX_HASH_VALUE 77
/* maximum key range = 73, duplicates = 0 */
static inline unsigned int
hash (register const char *str, register size_t len)
{
static const unsigned char asso_values[] =
{
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 5, 78, 39, 14, 1,
31, 31, 13, 13, 78, 78, 22, 25, 10, 2,
7, 78, 22, 0, 1, 3, 1, 78, 0, 36,
14, 17, 20, 78, 78, 78, 78, 5, 78, 39,
14, 1, 31, 31, 13, 13, 78, 78, 22, 25,
10, 2, 7, 78, 22, 0, 1, 3, 1, 78,
0, 36, 14, 17, 20, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78
};
register unsigned int hval = len;
switch (hval)
{
default:
hval += asso_values[(unsigned char)str[9]];
/*FALLTHROUGH*/
case 9:
case 8:
case 7:
case 6:
case 5:
case 4:
case 3:
case 2:
case 1:
hval += asso_values[(unsigned char)str[0]+2];
break;
}
return hval + asso_values[(unsigned char)str[len - 1]];
}
const StringReplacement *
gumbo_get_svg_attr_replacement (register const char *str, register size_t len)
{
static const unsigned char lengthtable[] =
{
0, 0, 0, 0, 0, 4, 0, 7, 7, 0, 8, 9, 10, 11,
11, 11, 11, 10, 16, 18, 16, 12, 16, 11, 13, 11, 12, 11,
16, 0, 17, 9, 9, 8, 9, 10, 13, 10, 12, 14, 8, 4,
12, 19, 7, 9, 12, 12, 11, 14, 10, 19, 8, 16, 13, 16,
16, 15, 10, 12, 0, 0, 13, 13, 13, 0, 0, 9, 16, 0,
0, 0, 0, 0, 0, 0, 0, 17
};
static const StringReplacement wordlist[] =
{
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0},
{"refx", "refX"},
{(char*)0,(char*)0},
{"viewbox", "viewBox"},
{"targetx", "targetX"},
{(char*)0,(char*)0},
{"calcmode", "calcMode"},
{"maskunits", "maskUnits"},
{"viewtarget", "viewTarget"},
{"tablevalues", "tableValues"},
{"markerunits", "markerUnits"},
{"stitchtiles", "stitchTiles"},
{"startoffset", "startOffset"},
{"numoctaves", "numOctaves"},
{"requiredfeatures", "requiredFeatures"},
{"requiredextensions", "requiredExtensions"},
{"specularexponent", "specularExponent"},
{"surfacescale", "surfaceScale"},
{"specularconstant", "specularConstant"},
{"repeatcount", "repeatCount"},
{"clippathunits", "clipPathUnits"},
{"filterunits", "filterUnits"},
{"lengthadjust", "lengthAdjust"},
{"markerwidth", "markerWidth"},
{"maskcontentunits", "maskContentUnits"},
{(char*)0,(char*)0},
{"limitingconeangle", "limitingConeAngle"},
{"pointsatx", "pointsAtX"},
{"repeatdur", "repeatDur"},
{"keytimes", "keyTimes"},
{"keypoints", "keyPoints"},
{"keysplines", "keySplines"},
{"gradientunits", "gradientUnits"},
{"textlength", "textLength"},
{"stddeviation", "stdDeviation"},
{"primitiveunits", "primitiveUnits"},
{"edgemode", "edgeMode"},
{"refy", "refY"},
{"spreadmethod", "spreadMethod"},
{"preserveaspectratio", "preserveAspectRatio"},
{"targety", "targetY"},
{"pointsatz", "pointsAtZ"},
{"markerheight", "markerHeight"},
{"patternunits", "patternUnits"},
{"baseprofile", "baseProfile"},
{"systemlanguage", "systemLanguage"},
{"zoomandpan", "zoomAndPan"},
{"patterncontentunits", "patternContentUnits"},
{"glyphref", "glyphRef"},
{"xchannelselector", "xChannelSelector"},
{"attributetype", "attributeType"},
{"kernelunitlength", "kernelUnitLength"},
{"ychannelselector", "yChannelSelector"},
{"diffuseconstant", "diffuseConstant"},
{"pathlength", "pathLength"},
{"kernelmatrix", "kernelMatrix"},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{"preservealpha", "preserveAlpha"},
{"attributename", "attributeName"},
{"basefrequency", "baseFrequency"},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{"pointsaty", "pointsAtY"},
{"patterntransform", "patternTransform"},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{"gradienttransform", "gradientTransform"}
};
if (len <= MAX_WORD_LENGTH && len >= MIN_WORD_LENGTH)
{
register unsigned int key = hash (str, len);
if (key <= MAX_HASH_VALUE)
if (len == lengthtable[key])
{
register const char *s = wordlist[key].from;
if (s && (((unsigned char)*str ^ (unsigned char)*s) & ~32) == 0 && !gumbo_ascii_strncasecmp(str, s, len))
return &wordlist[key];
}
}
return 0;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/svg_tags.c
Vendored
+137
View File
@@ -0,0 +1,137 @@
/* ANSI-C code produced by gperf version 3.1 */
/* Command-line: gperf -m100 src/svg_tags.gperf */
/* Computed positions: -k'3,7' */
/* Filtered by: gperf-filter.sed */
#include "replacement.h"
#include "macros.h"
#include "ascii.h"
#include <string.h>
#define TOTAL_KEYWORDS 36
#define MIN_WORD_LENGTH 6
#define MAX_WORD_LENGTH 19
#define MIN_HASH_VALUE 6
#define MAX_HASH_VALUE 42
/* maximum key range = 37, duplicates = 0 */
static inline unsigned int
hash (register const char *str, register size_t len)
{
static const unsigned char asso_values[] =
{
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 12, 2, 10, 22,
1, 28, 15, 1, 43, 43, 43, 0, 9, 26,
3, 17, 1, 11, 0, 22, 5, 43, 3, 2,
43, 43, 43, 43, 43, 43, 43, 43, 12, 2,
10, 22, 1, 28, 15, 1, 43, 43, 43, 0,
9, 26, 3, 17, 1, 11, 0, 22, 5, 43,
3, 2, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43
};
register unsigned int hval = len;
switch (hval)
{
default:
hval += asso_values[(unsigned char)str[6]+1];
/*FALLTHROUGH*/
case 6:
case 5:
case 4:
case 3:
hval += asso_values[(unsigned char)str[2]];
break;
}
return hval;
}
const StringReplacement *
gumbo_get_svg_tag_replacement (register const char *str, register size_t len)
{
static const unsigned char lengthtable[] =
{
0, 0, 0, 0, 0, 0, 6, 0, 7, 7, 7, 8, 11, 12,
12, 13, 11, 12, 16, 7, 7, 16, 11, 7, 19, 8, 13, 17,
11, 12, 7, 8, 17, 8, 18, 8, 14, 12, 14, 14, 13, 7,
14
};
static const StringReplacement wordlist[] =
{
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{(char*)0,(char*)0}, {(char*)0,(char*)0},
{"fetile", "feTile"},
{(char*)0,(char*)0},
{"femerge", "feMerge"},
{"feimage", "feImage"},
{"fefuncb", "feFuncB"},
{"glyphref", "glyphRef"},
{"femergenode", "feMergeNode"},
{"femorphology", "feMorphology"},
{"animatecolor", "animateColor"},
{"animatemotion", "animateMotion"},
{"fecomposite", "feComposite"},
{"feturbulence", "feTurbulence"},
{"animatetransform", "animateTransform"},
{"fefuncr", "feFuncR"},
{"fefunca", "feFuncA"},
{"feconvolvematrix", "feConvolveMatrix"},
{"fespotlight", "feSpotLight"},
{"fefuncg", "feFuncG"},
{"fecomponenttransfer", "feComponentTransfer"},
{"altglyph", "altGlyph"},
{"fecolormatrix", "feColorMatrix"},
{"fedisplacementmap", "feDisplacementMap"},
{"altglyphdef", "altGlyphDef"},
{"altglyphitem", "altGlyphItem"},
{"feflood", "feFlood"},
{"clippath", "clipPath"},
{"fediffuselighting", "feDiffuseLighting"},
{"textpath", "textPath"},
{"fespecularlighting", "feSpecularLighting"},
{"feoffset", "feOffset"},
{"fedistantlight", "feDistantLight"},
{"fepointlight", "fePointLight"},
{"lineargradient", "linearGradient"},
{"radialgradient", "radialGradient"},
{"foreignobject", "foreignObject"},
{"feblend", "feBlend"},
{"fegaussianblur", "feGaussianBlur"}
};
if (len <= MAX_WORD_LENGTH && len >= MIN_WORD_LENGTH)
{
register unsigned int key = hash (str, len);
if (key <= MAX_HASH_VALUE)
if (len == lengthtable[key])
{
register const char *s = wordlist[key].from;
if (s && (((unsigned char)*str ^ (unsigned char)*s) & ~32) == 0 && !gumbo_ascii_strncasecmp(str, s, len))
return &wordlist[key];
}
}
return 0;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/tag.c
Vendored
+223
View File
@@ -0,0 +1,223 @@
/*
Copyright 2011 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "nokogiri_gumbo.h"
#include "util.h"
#include "tag_lookup.h"
#include <assert.h>
#include <string.h>
static const char kGumboTagNames[GUMBO_TAG_LAST+1][15] = {
[GUMBO_TAG_HTML] = "html",
[GUMBO_TAG_HEAD] = "head",
[GUMBO_TAG_TITLE] = "title",
[GUMBO_TAG_BASE] = "base",
[GUMBO_TAG_LINK] = "link",
[GUMBO_TAG_META] = "meta",
[GUMBO_TAG_STYLE] = "style",
[GUMBO_TAG_SCRIPT] = "script",
[GUMBO_TAG_NOSCRIPT] = "noscript",
[GUMBO_TAG_TEMPLATE] = "template",
[GUMBO_TAG_BODY] = "body",
[GUMBO_TAG_ARTICLE] = "article",
[GUMBO_TAG_SECTION] = "section",
[GUMBO_TAG_NAV] = "nav",
[GUMBO_TAG_ASIDE] = "aside",
[GUMBO_TAG_H1] = "h1",
[GUMBO_TAG_H2] = "h2",
[GUMBO_TAG_H3] = "h3",
[GUMBO_TAG_H4] = "h4",
[GUMBO_TAG_H5] = "h5",
[GUMBO_TAG_H6] = "h6",
[GUMBO_TAG_HGROUP] = "hgroup",
[GUMBO_TAG_HEADER] = "header",
[GUMBO_TAG_FOOTER] = "footer",
[GUMBO_TAG_ADDRESS] = "address",
[GUMBO_TAG_P] = "p",
[GUMBO_TAG_HR] = "hr",
[GUMBO_TAG_PRE] = "pre",
[GUMBO_TAG_BLOCKQUOTE] = "blockquote",
[GUMBO_TAG_OL] = "ol",
[GUMBO_TAG_UL] = "ul",
[GUMBO_TAG_LI] = "li",
[GUMBO_TAG_DL] = "dl",
[GUMBO_TAG_DT] = "dt",
[GUMBO_TAG_DD] = "dd",
[GUMBO_TAG_FIGURE] = "figure",
[GUMBO_TAG_FIGCAPTION] = "figcaption",
[GUMBO_TAG_MAIN] = "main",
[GUMBO_TAG_DIV] = "div",
[GUMBO_TAG_A] = "a",
[GUMBO_TAG_EM] = "em",
[GUMBO_TAG_STRONG] = "strong",
[GUMBO_TAG_SMALL] = "small",
[GUMBO_TAG_S] = "s",
[GUMBO_TAG_CITE] = "cite",
[GUMBO_TAG_Q] = "q",
[GUMBO_TAG_DFN] = "dfn",
[GUMBO_TAG_ABBR] = "abbr",
[GUMBO_TAG_DATA] = "data",
[GUMBO_TAG_TIME] = "time",
[GUMBO_TAG_CODE] = "code",
[GUMBO_TAG_VAR] = "var",
[GUMBO_TAG_SAMP] = "samp",
[GUMBO_TAG_KBD] = "kbd",
[GUMBO_TAG_SUB] = "sub",
[GUMBO_TAG_SUP] = "sup",
[GUMBO_TAG_I] = "i",
[GUMBO_TAG_B] = "b",
[GUMBO_TAG_U] = "u",
[GUMBO_TAG_MARK] = "mark",
[GUMBO_TAG_RUBY] = "ruby",
[GUMBO_TAG_RT] = "rt",
[GUMBO_TAG_RP] = "rp",
[GUMBO_TAG_BDI] = "bdi",
[GUMBO_TAG_BDO] = "bdo",
[GUMBO_TAG_SPAN] = "span",
[GUMBO_TAG_BR] = "br",
[GUMBO_TAG_WBR] = "wbr",
[GUMBO_TAG_INS] = "ins",
[GUMBO_TAG_DEL] = "del",
[GUMBO_TAG_IMAGE] = "image",
[GUMBO_TAG_IMG] = "img",
[GUMBO_TAG_IFRAME] = "iframe",
[GUMBO_TAG_EMBED] = "embed",
[GUMBO_TAG_OBJECT] = "object",
[GUMBO_TAG_PARAM] = "param",
[GUMBO_TAG_VIDEO] = "video",
[GUMBO_TAG_AUDIO] = "audio",
[GUMBO_TAG_SOURCE] = "source",
[GUMBO_TAG_TRACK] = "track",
[GUMBO_TAG_CANVAS] = "canvas",
[GUMBO_TAG_MAP] = "map",
[GUMBO_TAG_AREA] = "area",
[GUMBO_TAG_MATH] = "math",
[GUMBO_TAG_MI] = "mi",
[GUMBO_TAG_MO] = "mo",
[GUMBO_TAG_MN] = "mn",
[GUMBO_TAG_MS] = "ms",
[GUMBO_TAG_MTEXT] = "mtext",
[GUMBO_TAG_MGLYPH] = "mglyph",
[GUMBO_TAG_MALIGNMARK] = "malignmark",
[GUMBO_TAG_ANNOTATION_XML] = "annotation-xml",
[GUMBO_TAG_SVG] = "svg",
[GUMBO_TAG_FOREIGNOBJECT] = "foreignobject",
[GUMBO_TAG_DESC] = "desc",
[GUMBO_TAG_TABLE] = "table",
[GUMBO_TAG_CAPTION] = "caption",
[GUMBO_TAG_COLGROUP] = "colgroup",
[GUMBO_TAG_COL] = "col",
[GUMBO_TAG_TBODY] = "tbody",
[GUMBO_TAG_THEAD] = "thead",
[GUMBO_TAG_TFOOT] = "tfoot",
[GUMBO_TAG_TR] = "tr",
[GUMBO_TAG_TD] = "td",
[GUMBO_TAG_TH] = "th",
[GUMBO_TAG_FORM] = "form",
[GUMBO_TAG_FIELDSET] = "fieldset",
[GUMBO_TAG_LEGEND] = "legend",
[GUMBO_TAG_LABEL] = "label",
[GUMBO_TAG_INPUT] = "input",
[GUMBO_TAG_BUTTON] = "button",
[GUMBO_TAG_SELECT] = "select",
[GUMBO_TAG_DATALIST] = "datalist",
[GUMBO_TAG_OPTGROUP] = "optgroup",
[GUMBO_TAG_OPTION] = "option",
[GUMBO_TAG_TEXTAREA] = "textarea",
[GUMBO_TAG_KEYGEN] = "keygen",
[GUMBO_TAG_OUTPUT] = "output",
[GUMBO_TAG_PROGRESS] = "progress",
[GUMBO_TAG_METER] = "meter",
[GUMBO_TAG_DETAILS] = "details",
[GUMBO_TAG_SUMMARY] = "summary",
[GUMBO_TAG_MENU] = "menu",
[GUMBO_TAG_MENUITEM] = "menuitem",
[GUMBO_TAG_APPLET] = "applet",
[GUMBO_TAG_ACRONYM] = "acronym",
[GUMBO_TAG_BGSOUND] = "bgsound",
[GUMBO_TAG_DIR] = "dir",
[GUMBO_TAG_FRAME] = "frame",
[GUMBO_TAG_FRAMESET] = "frameset",
[GUMBO_TAG_NOFRAMES] = "noframes",
[GUMBO_TAG_LISTING] = "listing",
[GUMBO_TAG_XMP] = "xmp",
[GUMBO_TAG_NEXTID] = "nextid",
[GUMBO_TAG_NOEMBED] = "noembed",
[GUMBO_TAG_PLAINTEXT] = "plaintext",
[GUMBO_TAG_RB] = "rb",
[GUMBO_TAG_STRIKE] = "strike",
[GUMBO_TAG_BASEFONT] = "basefont",
[GUMBO_TAG_BIG] = "big",
[GUMBO_TAG_BLINK] = "blink",
[GUMBO_TAG_CENTER] = "center",
[GUMBO_TAG_FONT] = "font",
[GUMBO_TAG_MARQUEE] = "marquee",
[GUMBO_TAG_MULTICOL] = "multicol",
[GUMBO_TAG_NOBR] = "nobr",
[GUMBO_TAG_SPACER] = "spacer",
[GUMBO_TAG_TT] = "tt",
[GUMBO_TAG_RTC] = "rtc",
[GUMBO_TAG_DIALOG] = "dialog",
[GUMBO_TAG_SEARCH] = "search",
[GUMBO_TAG_UNKNOWN] = "",
[GUMBO_TAG_LAST] = "",
};
const char* gumbo_normalized_tagname(GumboTag tag) {
assert(tag <= GUMBO_TAG_LAST);
const char *tagname = kGumboTagNames[tag];
assert(tagname);
return tagname;
}
void gumbo_tag_from_original_text(GumboStringPiece* text) {
if (text->data == NULL) {
return;
}
assert(text->length >= 2);
assert(text->data[0] == '<');
assert(text->data[text->length - 1] == '>');
if (text->data[1] == '/') {
// End tag
assert(text->length >= 3);
text->data += 2; // Move past </
text->length -= 3;
} else {
// Start tag
text->data += 1; // Move past <
text->length -= 2;
for (const char* c = text->data; c != text->data + text->length; ++c) {
switch (*c) {
case '\t':
case '\n':
case '\f':
case ' ':
case '/':
text->length = c - text->data;
return;
}
}
}
}
GumboTag gumbo_tagn_enum(const char *tagname, size_t tagname_length) {
const TagHashSlot *slot = gumbo_tag_lookup(tagname, tagname_length);
return slot ? slot->tag : GUMBO_TAG_UNKNOWN;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/tag_lookup.c
Vendored
+382
View File
@@ -0,0 +1,382 @@
/* ANSI-C code produced by gperf version 3.1 */
/* Command-line: gperf -m100 src/tag_lookup.gperf */
/* Computed positions: -k'1-2,$' */
/* Filtered by: gperf-filter.sed */
#include "tag_lookup.h"
#include "macros.h"
#include "ascii.h"
#include <string.h>
#define TOTAL_KEYWORDS 151
#define MIN_WORD_LENGTH 1
#define MAX_WORD_LENGTH 14
#define MIN_HASH_VALUE 9
#define MAX_HASH_VALUE 271
/* maximum key range = 263, duplicates = 0 */
static inline unsigned int
hash (register const char *str, register size_t len)
{
static const unsigned short asso_values[] =
{
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 11,
7, 6, 4, 4, 3, 4, 3, 3, 272, 272,
272, 272, 272, 272, 272, 70, 83, 152, 7, 16,
61, 98, 5, 76, 102, 126, 12, 19, 54, 54,
31, 97, 3, 4, 9, 33, 136, 113, 86, 15,
272, 272, 272, 272, 272, 272, 272, 70, 83, 152,
7, 16, 61, 98, 5, 76, 102, 126, 12, 19,
54, 54, 31, 97, 3, 4, 9, 33, 136, 113,
86, 15, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272, 272,
272, 272, 272, 272, 272, 272, 272, 272, 272
};
register unsigned int hval = len;
switch (hval)
{
default:
hval += asso_values[(unsigned char)str[1]+3];
/*FALLTHROUGH*/
case 1:
hval += asso_values[(unsigned char)str[0]];
break;
}
return hval + asso_values[(unsigned char)str[len - 1]];
}
const TagHashSlot *
gumbo_tag_lookup (register const char *str, register size_t len)
{
static const unsigned char lengthtable[] =
{
0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2,
2, 2, 2, 6, 2, 6, 6, 4, 2, 7, 6, 3, 0, 3,
0, 6, 6, 8, 5, 0, 0, 4, 5, 5, 8, 0, 2, 4,
5, 2, 0, 5, 4, 2, 0, 7, 0, 8, 5, 0, 0, 0,
0, 0, 0, 5, 3, 4, 5, 1, 4, 0, 4, 1, 2, 8,
7, 7, 6, 6, 8, 2, 8, 4, 2, 0, 6, 0, 0, 3,
4, 6, 13, 4, 4, 6, 8, 0, 8, 4, 0, 6, 0, 8,
4, 5, 0, 2, 2, 9, 2, 4, 0, 8, 4, 2, 4, 8,
7, 0, 2, 5, 2, 0, 6, 0, 3, 2, 2, 6, 3, 8,
7, 2, 5, 7, 0, 2, 6, 2, 4, 3, 0, 10, 5, 6,
3, 1, 2, 0, 6, 0, 5, 5, 0, 3, 0, 3, 3, 1,
4, 6, 4, 7, 3, 0, 0, 2, 10, 10, 0, 0, 6, 1,
4, 6, 3, 0, 2, 5, 6, 4, 3, 4, 0, 7, 3, 0,
0, 0, 4, 0, 0, 5, 0, 0, 0, 6, 0, 14, 8, 1,
3, 0, 0, 7, 3, 0, 0, 0, 0, 0, 0, 5, 3, 0,
0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 7, 6, 0, 0,
0, 0, 0, 5, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0,
3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0,
0, 0, 5, 0, 0, 3
};
static const TagHashSlot wordlist[] =
{
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"s", GUMBO_TAG_S},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"h6", GUMBO_TAG_H6},
{"h5", GUMBO_TAG_H5},
{"h4", GUMBO_TAG_H4},
{"h3", GUMBO_TAG_H3},
{"spacer", GUMBO_TAG_SPACER},
{"h2", GUMBO_TAG_H2},
{"header", GUMBO_TAG_HEADER},
{"search", GUMBO_TAG_SEARCH},
{"head", GUMBO_TAG_HEAD},
{"h1", GUMBO_TAG_H1},
{"details", GUMBO_TAG_DETAILS},
{"select", GUMBO_TAG_SELECT},
{"dir", GUMBO_TAG_DIR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"del", GUMBO_TAG_DEL},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"source", GUMBO_TAG_SOURCE},
{"legend", GUMBO_TAG_LEGEND},
{"datalist", GUMBO_TAG_DATALIST},
{"meter", GUMBO_TAG_METER},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"math", GUMBO_TAG_MATH},
{"label", GUMBO_TAG_LABEL},
{"table", GUMBO_TAG_TABLE},
{"template", GUMBO_TAG_TEMPLATE},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"rp", GUMBO_TAG_RP},
{"time", GUMBO_TAG_TIME},
{"title", GUMBO_TAG_TITLE},
{"hr", GUMBO_TAG_HR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"tbody", GUMBO_TAG_TBODY},
{"samp", GUMBO_TAG_SAMP},
{"tr", GUMBO_TAG_TR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"marquee", GUMBO_TAG_MARQUEE},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"menuitem", GUMBO_TAG_MENUITEM},
{"small", GUMBO_TAG_SMALL},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"embed", GUMBO_TAG_EMBED},
{"map", GUMBO_TAG_MAP},
{"menu", GUMBO_TAG_MENU},
{"param", GUMBO_TAG_PARAM},
{"p", GUMBO_TAG_P},
{"nobr", GUMBO_TAG_NOBR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"span", GUMBO_TAG_SPAN},
{"u", GUMBO_TAG_U},
{"em", GUMBO_TAG_EM},
{"noframes", GUMBO_TAG_NOFRAMES},
{"section", GUMBO_TAG_SECTION},
{"noembed", GUMBO_TAG_NOEMBED},
{"nextid", GUMBO_TAG_NEXTID},
{"footer", GUMBO_TAG_FOOTER},
{"noscript", GUMBO_TAG_NOSCRIPT},
{"dl", GUMBO_TAG_DL},
{"progress", GUMBO_TAG_PROGRESS},
{"font", GUMBO_TAG_FONT},
{"mo", GUMBO_TAG_MO},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"script", GUMBO_TAG_SCRIPT},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"pre", GUMBO_TAG_PRE},
{"main", GUMBO_TAG_MAIN},
{"object", GUMBO_TAG_OBJECT},
{"foreignobject", GUMBO_TAG_FOREIGNOBJECT},
{"form", GUMBO_TAG_FORM},
{"data", GUMBO_TAG_DATA},
{"applet", GUMBO_TAG_APPLET},
{"fieldset", GUMBO_TAG_FIELDSET},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"textarea", GUMBO_TAG_TEXTAREA},
{"abbr", GUMBO_TAG_ABBR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"figure", GUMBO_TAG_FIGURE},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"optgroup", GUMBO_TAG_OPTGROUP},
{"meta", GUMBO_TAG_META},
{"tfoot", GUMBO_TAG_TFOOT},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"ul", GUMBO_TAG_UL},
{"li", GUMBO_TAG_LI},
{"plaintext", GUMBO_TAG_PLAINTEXT},
{"rb", GUMBO_TAG_RB},
{"body", GUMBO_TAG_BODY},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"basefont", GUMBO_TAG_BASEFONT},
{"ruby", GUMBO_TAG_RUBY},
{"mi", GUMBO_TAG_MI},
{"base", GUMBO_TAG_BASE},
{"frameset", GUMBO_TAG_FRAMESET},
{"summary", GUMBO_TAG_SUMMARY},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"dd", GUMBO_TAG_DD},
{"frame", GUMBO_TAG_FRAME},
{"td", GUMBO_TAG_TD},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"option", GUMBO_TAG_OPTION},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"svg", GUMBO_TAG_SVG},
{"br", GUMBO_TAG_BR},
{"ol", GUMBO_TAG_OL},
{"dialog", GUMBO_TAG_DIALOG},
{"sup", GUMBO_TAG_SUP},
{"multicol", GUMBO_TAG_MULTICOL},
{"article", GUMBO_TAG_ARTICLE},
{"rt", GUMBO_TAG_RT},
{"image", GUMBO_TAG_IMAGE},
{"listing", GUMBO_TAG_LISTING},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"dt", GUMBO_TAG_DT},
{"mglyph", GUMBO_TAG_MGLYPH},
{"tt", GUMBO_TAG_TT},
{"html", GUMBO_TAG_HTML},
{"wbr", GUMBO_TAG_WBR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"figcaption", GUMBO_TAG_FIGCAPTION},
{"style", GUMBO_TAG_STYLE},
{"strike", GUMBO_TAG_STRIKE},
{"dfn", GUMBO_TAG_DFN},
{"a", GUMBO_TAG_A},
{"th", GUMBO_TAG_TH},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"hgroup", GUMBO_TAG_HGROUP},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"mtext", GUMBO_TAG_MTEXT},
{"thead", GUMBO_TAG_THEAD},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"var", GUMBO_TAG_VAR},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"xmp", GUMBO_TAG_XMP},
{"kbd", GUMBO_TAG_KBD},
{"i", GUMBO_TAG_I},
{"link", GUMBO_TAG_LINK},
{"output", GUMBO_TAG_OUTPUT},
{"mark", GUMBO_TAG_MARK},
{"acronym", GUMBO_TAG_ACRONYM},
{"div", GUMBO_TAG_DIV},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"ms", GUMBO_TAG_MS},
{"malignmark", GUMBO_TAG_MALIGNMARK},
{"blockquote", GUMBO_TAG_BLOCKQUOTE},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"center", GUMBO_TAG_CENTER},
{"b", GUMBO_TAG_B},
{"desc", GUMBO_TAG_DESC},
{"canvas", GUMBO_TAG_CANVAS},
{"col", GUMBO_TAG_COL},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"mn", GUMBO_TAG_MN},
{"track", GUMBO_TAG_TRACK},
{"iframe", GUMBO_TAG_IFRAME},
{"code", GUMBO_TAG_CODE},
{"sub", GUMBO_TAG_SUB},
{"area", GUMBO_TAG_AREA},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"address", GUMBO_TAG_ADDRESS},
{"ins", GUMBO_TAG_INS},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"cite", GUMBO_TAG_CITE},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"input", GUMBO_TAG_INPUT},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"keygen", GUMBO_TAG_KEYGEN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"annotation-xml", GUMBO_TAG_ANNOTATION_XML},
{"colgroup", GUMBO_TAG_COLGROUP},
{"q", GUMBO_TAG_Q},
{"big", GUMBO_TAG_BIG},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"bgsound", GUMBO_TAG_BGSOUND},
{"nav", GUMBO_TAG_NAV},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"video", GUMBO_TAG_VIDEO},
{"img", GUMBO_TAG_IMG},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"audio", GUMBO_TAG_AUDIO},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"caption", GUMBO_TAG_CAPTION},
{"strong", GUMBO_TAG_STRONG},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"aside", GUMBO_TAG_ASIDE},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"button", GUMBO_TAG_BUTTON},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"bdo", GUMBO_TAG_BDO},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"bdi", GUMBO_TAG_BDI},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"blink", GUMBO_TAG_BLINK},
{(char*)0,GUMBO_TAG_UNKNOWN},
{(char*)0,GUMBO_TAG_UNKNOWN},
{"rtc", GUMBO_TAG_RTC}
};
if (len <= MAX_WORD_LENGTH && len >= MIN_WORD_LENGTH)
{
register unsigned int key = hash (str, len);
if (key <= MAX_HASH_VALUE)
if (len == lengthtable[key])
{
register const char *s = wordlist[key].key;
if (s && (((unsigned char)*str ^ (unsigned char)*s) & ~32) == 0 && !gumbo_ascii_strncasecmp(str, s, len))
return &wordlist[key];
}
}
return 0;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/tag_lookup.h
Vendored
+13
View File
@@ -0,0 +1,13 @@
#ifndef GUMBO_TAG_LOOKUP_H_
#define GUMBO_TAG_LOOKUP_H_
#include "nokogiri_gumbo.h"
typedef struct {
const char *key;
const GumboTag tag;
} TagHashSlot;
const TagHashSlot *gumbo_tag_lookup(const char *str, size_t len);
#endif // GUMBO_TAG_LOOKUP_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/token_buffer.c
Vendored
+79
View File
@@ -0,0 +1,79 @@
/*
Copyright 2018 Stephen Checkoway
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <assert.h>
#include "ascii.h"
#include "token_buffer.h"
#include "tokenizer.h"
#include "util.h"
struct GumboInternalCharacterToken {
GumboSourcePosition position;
GumboStringPiece original_text;
int c;
};
void gumbo_character_token_buffer_init(GumboCharacterTokenBuffer* buffer) {
buffer->data = NULL;
buffer->length = 0;
buffer->capacity = 0;
}
void gumbo_character_token_buffer_append (
const GumboToken* token,
GumboCharacterTokenBuffer* buffer
) {
assert(token->type == GUMBO_TOKEN_WHITESPACE
|| token->type == GUMBO_TOKEN_CHARACTER);
if (buffer->length == buffer->capacity) {
if (buffer->capacity == 0)
buffer->capacity = 10;
else
buffer->capacity *= 2;
size_t bytes = sizeof(*buffer->data) * buffer->capacity;
buffer->data = gumbo_realloc(buffer->data, bytes);
}
size_t index = buffer->length++;
buffer->data[index].position = token->position;
buffer->data[index].original_text = token->original_text;
buffer->data[index].c = token->v.character;
}
void gumbo_character_token_buffer_get (
const GumboCharacterTokenBuffer* buffer,
size_t index,
struct GumboInternalToken* output
) {
assert(index < buffer->length);
int c = buffer->data[index].c;
output->type = gumbo_ascii_isspace(c)?
GUMBO_TOKEN_WHITESPACE : GUMBO_TOKEN_CHARACTER;
output->position = buffer->data[index].position;
output->original_text = buffer->data[index].original_text;
output->v.character = c;
}
void gumbo_character_token_buffer_clear(GumboCharacterTokenBuffer* buffer) {
buffer->length = 0;
}
void gumbo_character_token_buffer_destroy(GumboCharacterTokenBuffer* buffer) {
gumbo_free(buffer->data);
buffer->data = NULL;
buffer->length = 0;
buffer->capacity = 0;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/token_buffer.h
Vendored
+71
View File
@@ -0,0 +1,71 @@
/*
Copyright 2018 Stephen Checkoway
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#ifndef GUMBO_TOKEN_BUFFER_H
#define GUMBO_TOKEN_BUFFER_H
#include <stdbool.h>
#include <stddef.h>
#include "nokogiri_gumbo.h"
#ifdef __cplusplus
extern "C" {
#endif
struct GumboInternalCharacterToken;
struct GumboInternalToken;
// A struct representing a growable sequence of character (and whitespace)
// tokens.
typedef struct {
// A pointer to the start of the sequence.
struct GumboInternalCharacterToken* data;
// The length of the sequence.
size_t length;
// The capacity of the buffer.
size_t capacity;
} GumboCharacterTokenBuffer;
// Initializes a new GumboCharacterTokenBuffer.
void gumbo_character_token_buffer_init(GumboCharacterTokenBuffer* buffer);
// Appends a character (or whitespace) token.
void gumbo_character_token_buffer_append (
const struct GumboInternalToken* token,
GumboCharacterTokenBuffer* buffer
);
void gumbo_character_token_buffer_get (
const GumboCharacterTokenBuffer* buffer,
size_t index,
struct GumboInternalToken* output
);
// Reinitialize this string buffer. This clears it by setting length=0. It
// does not zero out the buffer itself.
void gumbo_character_token_buffer_clear(GumboCharacterTokenBuffer* buffer);
// Deallocates this GumboCharacterTokenBuffer.
void gumbo_character_token_buffer_destroy(GumboCharacterTokenBuffer* buffer);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_TOKEN_BUFFER_H
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/token_type.h
Vendored
+17
View File
@@ -0,0 +1,17 @@
#ifndef GUMBO_TOKEN_TYPE_H_
#define GUMBO_TOKEN_TYPE_H_
// An enum representing the type of token.
typedef enum {
GUMBO_TOKEN_DOCTYPE,
GUMBO_TOKEN_START_TAG,
GUMBO_TOKEN_END_TAG,
GUMBO_TOKEN_COMMENT,
GUMBO_TOKEN_WHITESPACE,
GUMBO_TOKEN_CHARACTER,
GUMBO_TOKEN_CDATA,
GUMBO_TOKEN_NULL,
GUMBO_TOKEN_EOF
} GumboTokenType;
#endif // GUMBO_TOKEN_TYPE_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/tokenizer.c
Vendored
+3494
View File
File diff suppressed because it is too large Load Diff
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/tokenizer.h
Vendored
+112
View File
@@ -0,0 +1,112 @@
#ifndef GUMBO_TOKENIZER_H_
#define GUMBO_TOKENIZER_H_
// This contains an implementation of a tokenizer for HTML5. It consumes a
// buffer of UTF-8 characters, and then emits a stream of tokens.
#include <stdbool.h>
#include <stddef.h>
#include "nokogiri_gumbo.h"
#include "token_type.h"
#include "tokenizer_states.h"
#ifdef __cplusplus
extern "C" {
#endif
struct GumboInternalParser;
// Struct containing all information pertaining to doctype tokens.
typedef struct GumboInternalTokenDocType {
const char* name;
const char* public_identifier;
const char* system_identifier;
bool force_quirks;
// There's no way to tell a 0-length public or system ID apart from the
// absence of a public or system ID, but they're handled different by the
// spec, so we need bool flags for them.
bool has_public_identifier;
bool has_system_identifier;
} GumboTokenDocType;
// Struct containing all information pertaining to start tag tokens.
typedef struct GumboInternalTokenStartTag {
GumboTag tag;
// NULL unless tag is GUMBO_TAG_UNKNOWN
char *name;
GumboVector /* GumboAttribute */ attributes;
bool is_self_closing;
} GumboTokenStartTag;
// Struct containing all information pertaining to end tag tokens.
typedef struct GumboInternalTokenEndTag {
GumboTag tag;
// NULL unless tag is GUMBO_TAG_UNKNOWN
char *name;
} GumboTokenEndTag;
// A data structure representing a single token in the input stream. This
// contains an enum for the type, the source position, a GumboStringPiece
// pointing to the original text, and then a union for any parsed data.
typedef struct GumboInternalToken {
GumboTokenType type;
GumboSourcePosition position;
GumboStringPiece original_text;
union {
GumboTokenDocType doc_type;
GumboTokenStartTag start_tag;
GumboTokenEndTag end_tag;
const char* text; // For comments.
int character; // For character, whitespace, null, and EOF tokens.
} v;
} GumboToken;
// Initializes the tokenizer state within the GumboParser object, setting up a
// parse of the specified text.
void gumbo_tokenizer_state_init (
struct GumboInternalParser* parser,
const char* text,
size_t text_length
);
// Destroys the tokenizer state within the GumboParser object, freeing any
// dynamically-allocated structures within it.
void gumbo_tokenizer_state_destroy(struct GumboInternalParser* parser);
// Sets the tokenizer state to the specified value. This is needed by some
// parser states, which alter the state of the tokenizer in response to tags
// seen.
void gumbo_tokenizer_set_state (
struct GumboInternalParser* parser,
GumboTokenizerEnum state
);
// Flags whether the adjusted current node is a foreign content element. This
// is necessary for the markup declaration open state, where the tokenizer
// must be aware of the state of the parser to properly tokenize bad comment
// tags.
// https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
void gumbo_tokenizer_set_is_adjusted_current_node_foreign (
struct GumboInternalParser* parser,
bool is_foreign
);
// Lexes a single token from the specified buffer, filling the output with the
// parsed GumboToken data structure.
void gumbo_lex(struct GumboInternalParser* parser, GumboToken* output);
// Frees the internally-allocated pointers within a GumboToken. Note that this
// doesn't free the token itself, since oftentimes it will be allocated on the
// stack.
//
// Note that if you are handing over ownership of the internal strings to some
// other data structure - for example, a parse tree - these do not need to be
// freed.
void gumbo_token_destroy(GumboToken* token);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_TOKENIZER_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/tokenizer_states.h
Vendored
+339
View File
@@ -0,0 +1,339 @@
#ifndef GUMBO_TOKENIZER_STATES_H_
#define GUMBO_TOKENIZER_STATES_H_
// This contains the list of states used in the tokenizer. Although at first
// glance it seems like these could be kept internal to the tokenizer, several
// of the actions in the parser require that it reach into the tokenizer and
// reset the tokenizer state. For that to work, it needs to have the
// definitions of individual states available.
//
// This may also be useful for providing more detailed error messages for parse
// errors, as we can match up states and inputs in a table without having to
// clutter the tokenizer code with lots of precise error messages.
// The ordering of this enum is also used to build the dispatch table for the
// tokenizer state machine, so if it is changed, be sure to update that too.
typedef enum {
// 12.2.5.1 Data state
// https://html.spec.whatwg.org/multipage/parsing.html#data-state
GUMBO_LEX_DATA,
// 12.2.5.2 RCDATA state
// https://html.spec.whatwg.org/multipage/parsing.html#rcdata-state
GUMBO_LEX_RCDATA,
// 12.2.5.3 RAWTEXT state
// https://html.spec.whatwg.org/multipage/parsing.html#rawtext-state<Paste>
GUMBO_LEX_RAWTEXT,
// 12.2.5.4 Script data state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-state
GUMBO_LEX_SCRIPT_DATA,
// 12.2.5.5 PLAINTEXT state
// https://html.spec.whatwg.org/multipage/parsing.html#plaintext-state
GUMBO_LEX_PLAINTEXT,
// 12.2.5.6 Tag open state
// https://html.spec.whatwg.org/multipage/parsing.html#tag-open-state
GUMBO_LEX_TAG_OPEN,
// 12.2.5.7 End tag open state
// https://html.spec.whatwg.org/multipage/parsing.html#end-tag-open-state
GUMBO_LEX_END_TAG_OPEN,
// 12.2.5.8 Tag name state
// https://html.spec.whatwg.org/multipage/parsing.html#tag-name-state
GUMBO_LEX_TAG_NAME,
// 12.2.5.9 RCDATA less-than sign state
// https://html.spec.whatwg.org/multipage/parsing.html#rcdata-less-than-sign-state
GUMBO_LEX_RCDATA_LT,
// 12.2.5.10 RCDATA end tag open state
// https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-open-state
GUMBO_LEX_RCDATA_END_TAG_OPEN,
// 12.2.5.11 RCDATA end tag name state
// https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-name-state
GUMBO_LEX_RCDATA_END_TAG_NAME,
// 12.2.5.12 RAWTEXT less-than sign state
// https://html.spec.whatwg.org/multipage/parsing.html#rawtext-less-than-sign-state
GUMBO_LEX_RAWTEXT_LT,
// 12.2.5.13 RAWTEXT end tag open state
// https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-open-state
GUMBO_LEX_RAWTEXT_END_TAG_OPEN,
// 12.2.5.14 RAWTEXT end tag name state
// https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-name-state
GUMBO_LEX_RAWTEXT_END_TAG_NAME,
// 12.2.5.15 Script data less-than sign state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-less-than-sign-state
GUMBO_LEX_SCRIPT_DATA_LT,
// 12.2.5.16 Script data end tag open state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-open-state
GUMBO_LEX_SCRIPT_DATA_END_TAG_OPEN,
// 12.2.5.17 Script data end tag name state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-name-state
GUMBO_LEX_SCRIPT_DATA_END_TAG_NAME,
// 12.2.5.18 Script data escape start state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_START,
// 12.2.5.19 Script data escape start dash state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-dash-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_START_DASH,
// 12.2.5.20 Script data escaped state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED,
// 12.2.5.21 Script data escaped dash state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_DASH,
// 12.2.5.22 Script data escaped dash dash state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-dash-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_DASH_DASH,
// 12.2.5.23 Script data escaped less than sign state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-less-than-sign-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_LT,
// 12.2.5.24 Script data escaped end tag open state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-open-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_END_TAG_OPEN,
// 12.2.5.25 Script data escaped end tag name state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-name-state
GUMBO_LEX_SCRIPT_DATA_ESCAPED_END_TAG_NAME,
// 12.2.5.26 Script data double escape start state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-start-state
GUMBO_LEX_SCRIPT_DATA_DOUBLE_ESCAPED_START,
// 12.2.5.27 Script data double escaped state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-state
GUMBO_LEX_SCRIPT_DATA_DOUBLE_ESCAPED,
// 12.2.5.28 Script data double escaped dash state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-dash-state
GUMBO_LEX_SCRIPT_DATA_DOUBLE_ESCAPED_DASH,
// 12.2.5.29 Script data double escaped dash dash state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-dash-dash-state
GUMBO_LEX_SCRIPT_DATA_DOUBLE_ESCAPED_DASH_DASH,
// 12.2.5.30 Script data double escaped less-than sign state
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-less-than-sign-state
GUMBO_LEX_SCRIPT_DATA_DOUBLE_ESCAPED_LT,
// 12.2.5.31 Script data double escape end state (XXX: spec bug with the
// name?)
// https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-end-state
GUMBO_LEX_SCRIPT_DATA_DOUBLE_ESCAPED_END,
// 12.2.5.32 Before attribute name state
// https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-name-state
GUMBO_LEX_BEFORE_ATTR_NAME,
// 12.2.5.33 Attributet name state
// https://html.spec.whatwg.org/multipage/parsing.html#attribute-name-state
GUMBO_LEX_ATTR_NAME,
// 12.2.5.34 After attribute name state
// https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-name-state
GUMBO_LEX_AFTER_ATTR_NAME,
// 12.2.5.35 Before attribute value state
// https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-value-state
GUMBO_LEX_BEFORE_ATTR_VALUE,
// 12.2.5.36 Attribute value (double-quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(double-quoted)-state
GUMBO_LEX_ATTR_VALUE_DOUBLE_QUOTED,
// 12.2.5.37 Attribute value (single-quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(single-quoted)-state
GUMBO_LEX_ATTR_VALUE_SINGLE_QUOTED,
// 12.2.5.38 Attribute value (unquoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(unquoted)-state
GUMBO_LEX_ATTR_VALUE_UNQUOTED,
// 12.2.5.39 After attribute value (quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-value-(quoted)-state
GUMBO_LEX_AFTER_ATTR_VALUE_QUOTED,
// 12.2.5.40 Self-closing start tag state
// https://html.spec.whatwg.org/multipage/parsing.html#self-closing-start-tag-state
GUMBO_LEX_SELF_CLOSING_START_TAG,
// 12.2.5.41 Bogus comment state
// https://html.spec.whatwg.org/multipage/parsing.html#bogus-comment-state
GUMBO_LEX_BOGUS_COMMENT,
// 12.2.5.42 Markup declaration open state
// https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
GUMBO_LEX_MARKUP_DECLARATION_OPEN,
// 12.2.5.43 Comment start state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state
GUMBO_LEX_COMMENT_START,
// 12.2.5.44 Comment start dash state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-start-dash-state
GUMBO_LEX_COMMENT_START_DASH,
// 12.2.5.45 Comment state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-state
GUMBO_LEX_COMMENT,
// 12.2.5.46 Comment less-than sign state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-state
GUMBO_LEX_COMMENT_LT,
// 12.2.5.47 Comment less-than sign bang state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-state
GUMBO_LEX_COMMENT_LT_BANG,
// 12.2.5.48 Comment less-than sign bang dash state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-state
GUMBO_LEX_COMMENT_LT_BANG_DASH,
// 12.2.5.49 Comment less-than sign bang dash dash state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-dash-state
GUMBO_LEX_COMMENT_LT_BANG_DASH_DASH,
// 12.2.5.50 Comment end dash state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-end-dash-state
GUMBO_LEX_COMMENT_END_DASH,
// 12.2.5.51 Comment end state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-end-state
GUMBO_LEX_COMMENT_END,
// 12.2.5.52 Comment end bang state
// https://html.spec.whatwg.org/multipage/parsing.html#comment-end-bang-state
GUMBO_LEX_COMMENT_END_BANG,
// 12.2.5.53 DOCTYPE state
// https://html.spec.whatwg.org/multipage/parsing.html#doctype-state
GUMBO_LEX_DOCTYPE,
// 12.2.5.54 Before DOCTYPE name state
// https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-name-state
GUMBO_LEX_BEFORE_DOCTYPE_NAME,
// 12.2.5.55 DOCTYPE name state
// https://html.spec.whatwg.org/multipage/parsing.html#doctype-name-state
GUMBO_LEX_DOCTYPE_NAME,
// 12.2.5.56 After DOCTYPE name state
// https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-name-state
GUMBO_LEX_AFTER_DOCTYPE_NAME,
// 12.2.5.57 After DOCTYPE public keyword state
// https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-keyword-state
GUMBO_LEX_AFTER_DOCTYPE_PUBLIC_KEYWORD,
// 12.2.5.58 Before DOCTYPE public identifier state
// https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-public-identifier-state
GUMBO_LEX_BEFORE_DOCTYPE_PUBLIC_ID,
// 12.2.5.59 DOCTYPE public identifier (double-quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(double-quoted)-state
GUMBO_LEX_DOCTYPE_PUBLIC_ID_DOUBLE_QUOTED,
// 12.2.5.60 DOCTYPE public identifier (single-quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(single-quoted)-state
GUMBO_LEX_DOCTYPE_PUBLIC_ID_SINGLE_QUOTED,
// 12.2.5.61 After DOCTYPE public identifier state
// https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-identifier-state
GUMBO_LEX_AFTER_DOCTYPE_PUBLIC_ID,
// 12.2.5.62 Between DOCTYPE public and system identifiers state
// https://html.spec.whatwg.org/multipage/parsing.html#between-doctype-public-and-system-identifiers-state
GUMBO_LEX_BETWEEN_DOCTYPE_PUBLIC_SYSTEM_ID,
// 12.2.5.63 After DOCTYPE system keyword state
// https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-keyword-state
GUMBO_LEX_AFTER_DOCTYPE_SYSTEM_KEYWORD,
// 12.2.5.64 Before DOCTYPE system identifier state
// https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-system-identifier-state
GUMBO_LEX_BEFORE_DOCTYPE_SYSTEM_ID,
// 12.2.5.65 DOCTYPE system identifier (double-quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(double-quoted)-state
GUMBO_LEX_DOCTYPE_SYSTEM_ID_DOUBLE_QUOTED,
// 12.2.5.66 DOCTYPE system identifier (single-quoted) state
// https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(single-quoted)-state
GUMBO_LEX_DOCTYPE_SYSTEM_ID_SINGLE_QUOTED,
// 12.2.5.67 After DOCTYPE system identifier state
// https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-identifier-state
GUMBO_LEX_AFTER_DOCTYPE_SYSTEM_ID,
// 12.2.5.68 Bogus DOCTYPE state
// https://html.spec.whatwg.org/multipage/parsing.html#bogus-doctype-state
GUMBO_LEX_BOGUS_DOCTYPE,
// 12.2.5.69 CDATA section state
// https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-state
GUMBO_LEX_CDATA_SECTION,
// 12.2.5.70 CDATA section bracket state
// https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-bracket-state
GUMBO_LEX_CDATA_SECTION_BRACKET,
// 12.2.5.71 CDATA section end state
// https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-end-state
GUMBO_LEX_CDATA_SECTION_END,
// 12.2.5.72 Character reference state
// https://html.spec.whatwg.org/multipage/parsing.html#character-reference-state
GUMBO_LEX_CHARACTER_REFERENCE,
// 12.2.5.73 Named character reference state
// https://html.spec.whatwg.org/multipage/parsing.html#named-character-reference-state
GUMBO_LEX_NAMED_CHARACTER_REFERENCE,
// 12.2.5.74 Ambiguous ampersand state
// https://html.spec.whatwg.org/multipage/parsing.html#ambiguous-ampersand-state
GUMBO_LEX_AMBIGUOUS_AMPERSAND,
// 12.2.5.75 Numeric character reference state
// https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-state
GUMBO_LEX_NUMERIC_CHARACTER_REFERENCE,
// 12.2.5.76 Hexadecimal character reference start state
// https://html.spec.whatwg.org/multipage/parsing.html#hexadecimal-character-reference-start-state
GUMBO_LEX_HEXADECIMAL_CHARACTER_REFERENCE_START,
// 12.2.5.77 Decimal character reference start state
// https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-start-state
GUMBO_LEX_DECIMAL_CHARACTER_REFERENCE_START,
// 12.2.5.78 Hexadecimal character reference state
// https://html.spec.whatwg.org/multipage/parsing.html#hexadecimal-character-reference-state
GUMBO_LEX_HEXADECIMAL_CHARACTER_REFERENCE,
// 12.2.5.79 Decimal character reference state
// https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-state
GUMBO_LEX_DECIMAL_CHARACTER_REFERENCE,
// 12.2.5.80 Numeric character reference end state
// https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state
GUMBO_LEX_NUMERIC_CHARACTER_REFERENCE_END
} GumboTokenizerEnum;
#endif // GUMBO_TOKENIZER_STATES_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/utf8.c
Vendored
+245
View File
@@ -0,0 +1,245 @@
/*
Copyright 2018 Craig Barnes.
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "utf8.h"
#include <assert.h>
#include <stdint.h>
#include <string.h>
#include "error.h"
#include "nokogiri_gumbo.h"
#include "parser.h"
#include "ascii.h"
#include "vector.h"
// References:
// * https://tools.ietf.org/html/rfc3629
// * https://html.spec.whatwg.org/multipage/parsing.html#preprocessing-the-input-stream
// The following code is a DFA-based UTF-8 decoder by Bjoern Hoehrmann.
// We wrap the inner table-based decoder routine in our own handling for
// newlines, tabs, invalid continuation bytes, and other conditions that
// the HTML5 spec fully specifies but normal UTF-8 decoders do not handle.
// See https://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.
// Copyright (c) 2008-2009 Bjoern Hoehrmann <bjoern@hoehrmann.de>
//
// Permission is hereby granted, free of charge, to any person obtaining a
// copy of this software and associated documentation files (the "Software"),
// to deal in the Software without restriction, including without limitation
// the rights to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
#define UTF8_ACCEPT 0
#define UTF8_REJECT 12
static const uint8_t utf8d[] = {
// The first part of the table maps bytes to character classes that
// to reduce the size of the transition table and create bitmasks.
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,
7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,
8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
10,3,3,3,3,3,3,3,3,3,3,3,3,4,3,3, 11,6,6,6,5,8,8,8,8,8,8,8,8,8,8,8,
// The second part is a transition table that maps a combination
// of a state of the automaton and a character class to a state.
0,12,24,36,60,96,84,12,12,12,48,72, 12,12,12,12,12,12,12,12,12,12,12,12,
12, 0,12,12,12,12,12, 0,12, 0,12,12, 12,24,12,12,12,12,12,24,12,24,12,12,
12,12,12,12,12,12,12,24,12,12,12,12, 12,24,12,12,12,12,12,12,12,24,12,12,
12,12,12,12,12,12,12,36,12,36,12,12, 12,36,12,12,12,12,12,36,12,36,12,12,
12,36,12,12,12,12,12,12,12,12,12,12,
};
static inline uint32_t decode(uint32_t* state, uint32_t* codep, uint32_t byte) {
uint32_t type = utf8d[byte];
*codep =
(*state != UTF8_ACCEPT)
? (byte & 0x3fu) | (*codep << 6)
: (0xff >> type) & (byte);
*state = utf8d[256 + *state + type];
return *state;
}
// END COPIED CODE.
// Adds a decoding error to the parser's error list, based on the current state
// of the Utf8Iterator.
static void add_error(Utf8Iterator* iter, GumboErrorType type) {
GumboParser* parser = iter->_parser;
GumboError* error = gumbo_add_error(parser);
if (!error) {
return;
}
error->type = type;
error->position = iter->_pos;
error->original_text.data = iter->_start;
error->original_text.length = iter->_width;
error->v.tokenizer.codepoint = iter->_current;
}
// Reads the next UTF-8 character in the iter.
// This assumes that iter->_start points to the beginning of the character.
// When this method returns, iter->_width and iter->_current will be set
// appropriately, as well as any error flags.
static void read_char(Utf8Iterator* iter) {
if (iter->_start >= iter->_end) {
// No input left to consume; emit an EOF and set width = 0.
iter->_current = -1;
iter->_width = 0;
return;
}
uint32_t code_point = 0;
uint32_t state = UTF8_ACCEPT;
for (const char* c = iter->_start; c < iter->_end; ++c) {
decode(&state, &code_point, (uint32_t)(unsigned char) (*c));
if (state == UTF8_ACCEPT) {
iter->_width = c - iter->_start + 1;
// This is the special handling for carriage returns that is mandated by
// the HTML5 spec. Since we're looking for particular 7-bit literal
// characters, we operate in terms of chars and only need a check for iter
// overrun, instead of having to read in a full next code point.
// https://html.spec.whatwg.org/multipage/parsing.html#preprocessing-the-input-stream
if (code_point == '\r') {
assert(iter->_width == 1);
const char* next = c + 1;
if (next < iter->_end && *next == '\n') {
// Advance the iter, as if the carriage return didn't exist.
++iter->_start;
// Preserve the true offset, since other tools that look at it may be
// unaware of HTML5's rules for converting \r into \n.
++iter->_pos.offset;
}
code_point = '\n';
}
iter->_current = code_point;
if (utf8_is_surrogate(code_point)) {
add_error(iter, GUMBO_ERR_SURROGATE_IN_INPUT_STREAM);
} else if (utf8_is_noncharacter(code_point)) {
add_error(iter, GUMBO_ERR_NONCHARACTER_IN_INPUT_STREAM);
} else if (utf8_is_control(code_point)
&& !(gumbo_ascii_isspace(code_point) || code_point == 0)) {
add_error(iter, GUMBO_ERR_CONTROL_CHARACTER_IN_INPUT_STREAM);
}
return;
} else if (state == UTF8_REJECT) {
// We don't want to consume the invalid continuation byte of a multi-byte
// run, but we do want to skip past an invalid first byte.
iter->_width = c - iter->_start + (c == iter->_start);
iter->_current = kUtf8ReplacementChar;
add_error(iter, GUMBO_ERR_UTF8_INVALID);
return;
}
}
// If we got here without exiting early, then we've reached the end of the
// iterator. Add an error for truncated input, set the width to consume the
// rest of the iterator, and emit a replacement character. The next time we
// enter this method, it will detect that there's no input to consume and
// output an EOF.
iter->_width = iter->_end - iter->_start;
iter->_current = kUtf8ReplacementChar;
add_error(iter, GUMBO_ERR_UTF8_TRUNCATED);
}
static void update_position(Utf8Iterator* iter) {
iter->_pos.offset += iter->_width;
if (iter->_current == '\n') {
++iter->_pos.line;
iter->_pos.column = 1;
} else if (iter->_current == '\t') {
int tab_stop = iter->_parser->_options->tab_stop;
iter->_pos.column = ((iter->_pos.column / tab_stop) + 1) * tab_stop;
} else if (iter->_current != -1) {
++iter->_pos.column;
}
}
void utf8iterator_init (
GumboParser* parser,
const char* source,
size_t source_length,
Utf8Iterator* iter
) {
iter->_start = source;
iter->_end = source + source_length;
iter->_pos.line = 1;
iter->_pos.column = 1;
iter->_pos.offset = 0;
iter->_parser = parser;
read_char(iter);
if (iter->_current == kUtf8BomChar) {
iter->_start += iter->_width;
iter->_pos.offset += iter->_width;
read_char(iter);
}
}
void utf8iterator_next(Utf8Iterator* iter) {
// We update positions based on the *last* character read, so that the first
// character following a newline is at column 1 in the next line.
update_position(iter);
iter->_start += iter->_width;
read_char(iter);
}
bool utf8iterator_maybe_consume_match (
Utf8Iterator* iter,
const char* prefix,
size_t length,
bool case_sensitive
) {
bool matched =
(iter->_start + length <= iter->_end)
&& (
case_sensitive
? !strncmp(iter->_start, prefix, length)
: !gumbo_ascii_strncasecmp(iter->_start, prefix, length)
)
;
if (matched) {
for (size_t i = 0; i < length; ++i) {
utf8iterator_next(iter);
}
return true;
} else {
return false;
}
}
void utf8iterator_mark(Utf8Iterator* iter) {
iter->_mark = iter->_start;
iter->_mark_pos = iter->_pos;
}
// Returns the current input stream position to the mark.
void utf8iterator_reset(Utf8Iterator* iter) {
iter->_start = iter->_mark;
iter->_pos = iter->_mark_pos;
read_char(iter);
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/utf8.h
Vendored
+164
View File
@@ -0,0 +1,164 @@
#ifndef GUMBO_UTF8_H_
#define GUMBO_UTF8_H_
// This contains an implementation of a UTF-8 iterator and decoder suitable for
// a HTML5 parser. This does a bit more than straight UTF-8 decoding. The
// HTML5 spec specifies that:
// 1. Decoding errors are parse errors.
// 2. Certain other codepoints (e.g. control characters) are parse errors.
// 3. Carriage returns and CR/LF groups are converted to line feeds.
// https://encoding.spec.whatwg.org/#utf-8-decode
//
// Also, we want to keep track of source positions for error handling. As a
// result, we fold all that functionality into this decoder, and can't use an
// off-the-shelf library.
//
// This header is internal-only, which is why we prefix functions with only
// utf8_ or utf8_iterator_ instead of gumbo_utf8_.
#include <stdbool.h>
#include <stddef.h>
#include "nokogiri_gumbo.h"
#include "macros.h"
#ifdef __cplusplus
extern "C" {
#endif
struct GumboInternalError;
struct GumboInternalParser;
// Unicode replacement char.
#define kUtf8ReplacementChar 0xFFFD
#define kUtf8BomChar 0xFEFF
#define kUtf8MaxChar 0x10FFFF
typedef struct GumboInternalUtf8Iterator {
// Points at the start of the code point most recently read into 'current'.
const char* _start;
// Points at the mark. The mark is initially set to the beginning of the
// input.
const char* _mark;
// Points past the end of the iter, like a past-the-end iterator in the STL.
const char* _end;
// The code point under the cursor.
int _current;
// The width in bytes of the current code point.
size_t _width;
// The SourcePosition for the current location.
GumboSourcePosition _pos;
// The SourcePosition for the mark.
GumboSourcePosition _mark_pos;
// Pointer back to the GumboParser instance, for configuration options and
// error recording.
struct GumboInternalParser* _parser;
} Utf8Iterator;
// Returns true if this Unicode code point is a surrogate.
CONST_FN static inline bool utf8_is_surrogate(int c) {
return c >= 0xD800 && c <= 0xDFFF;
}
// Returns true if this Unicode code point is a noncharacter.
CONST_FN static inline bool utf8_is_noncharacter(int c) {
return
(c >= 0xFDD0 && c <= 0xFDEF)
|| ((c & 0xFFFF) == 0xFFFE)
|| ((c & 0xFFFF) == 0xFFFF);
}
// Returns true if this Unicode code point is a control.
CONST_FN static inline bool utf8_is_control(int c) {
return ((unsigned int)c < 0x1Fu) || (c >= 0x7F && c <= 0x9F);
}
// Initializes a new Utf8Iterator from the given byte buffer. The source does
// not have to be NUL-terminated, but the length must be passed in explicitly.
void utf8iterator_init (
struct GumboInternalParser* parser,
const char* source,
size_t source_length,
Utf8Iterator* iter
);
// Advances the current position by one code point.
void utf8iterator_next(Utf8Iterator* iter);
// Returns the current code point as an integer.
static inline int utf8iterator_current(const Utf8Iterator* iter) {
return iter->_current;
}
// Retrieves and fills the output parameter with the current source position.
static inline void utf8iterator_get_position (
const Utf8Iterator* iter,
GumboSourcePosition* output
) {
*output = iter->_pos;
}
// Retrieves the marked position.
static inline GumboSourcePosition utf8iterator_get_mark_position (
const Utf8Iterator* iter
) {
return iter->_mark_pos;
}
// Retrieves a character pointer to the start of the current character.
static inline const char* utf8iterator_get_char_pointer(const Utf8Iterator* iter) {
return iter->_start;
}
// Retrieves the width of the current character.
static inline size_t utf8iterator_get_width(const Utf8Iterator* iter) {
return iter->_width;
}
// Retrieves a character pointer to 1 past the end of the buffer. This is
// necessary for certain state machines and string comparisons that would like
// to look directly for ASCII text in the buffer without going through the
// decoder.
static inline const char* utf8iterator_get_end_pointer(const Utf8Iterator* iter) {
return iter->_end;
}
// Retrieves a character pointer to the marked position.
static inline const char* utf8iterator_get_mark_pointer(const Utf8Iterator* iter) {
return iter->_mark;
}
// If the upcoming text in the buffer matches the specified prefix (which has
// length 'length'), consume it and return true. Otherwise, return false with
// no other effects. If the length of the string would overflow the buffer,
// this returns false. Note that prefix should not contain null bytes because
// of the use of strncmp/strncasecmp internally. All existing use-cases adhere
// to this.
bool utf8iterator_maybe_consume_match (
Utf8Iterator* iter,
const char* prefix,
size_t length,
bool case_sensitive
);
// "Marks" a particular location of interest in the input stream, so that it can
// later be reset() to. There's also the ability to record an error at the
// point that was marked, as oftentimes that's more useful than the last
// character before the error was detected.
void utf8iterator_mark(Utf8Iterator* iter);
// Returns the current input stream position to the mark.
void utf8iterator_reset(Utf8Iterator* iter);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_UTF8_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/util.c
Vendored
+66
View File
@@ -0,0 +1,66 @@
/*
Copyright 2017-2018 Craig Barnes.
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "util.h"
#include "nokogiri_gumbo.h"
void* gumbo_alloc(size_t size) {
void* ptr = malloc(size);
if (unlikely(ptr == NULL)) {
perror(__func__);
abort();
}
return ptr;
}
void* gumbo_realloc(void* ptr, size_t size) {
ptr = realloc(ptr, size);
if (unlikely(ptr == NULL)) {
perror(__func__);
abort();
}
return ptr;
}
void gumbo_free(void* ptr) {
free(ptr);
}
char* gumbo_strdup(const char* str) {
const size_t size = strlen(str) + 1;
// The strdup(3) function isn't available in strict "-std=c99" mode
// (it's part of POSIX, not C99), so use malloc(3) and memcpy(3)
// instead:
char* buffer = gumbo_alloc(size);
return memcpy(buffer, str, size);
}
#ifdef GUMBO_DEBUG
#include <stdarg.h>
// Debug function to trace operation of the parser
// (define GUMBO_DEBUG to use).
void gumbo_debug(const char* format, ...) {
va_list args;
va_start(args, format);
vprintf(format, args);
va_end(args);
fflush(stdout);
}
#endif
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/util.h
Vendored
+34
View File
@@ -0,0 +1,34 @@
#ifndef GUMBO_UTIL_H_
#define GUMBO_UTIL_H_
#include <stdbool.h>
#include <stddef.h>
#include "macros.h"
#ifdef __cplusplus
extern "C" {
#endif
// Utility function for allocating & copying a null-terminated string into a
// freshly-allocated buffer. This is necessary for proper memory management; we
// have the convention that all const char* in parse tree structures are
// freshly-allocated, so if we didn't copy, we'd try to delete a literal string
// when the parse tree is destroyed.
char* gumbo_strdup(const char* str) XMALLOC NONNULL_ARGS;
void* gumbo_alloc(size_t size) XMALLOC;
void* gumbo_realloc(void* ptr, size_t size) RETURNS_NONNULL;
void gumbo_free(void* ptr);
// Debug wrapper for printf
#ifdef GUMBO_DEBUG
void gumbo_debug(const char* format, ...) PRINTF(1);
#else
static inline void PRINTF(1) gumbo_debug(const char* UNUSED_ARG(format), ...) {};
#endif
#ifdef __cplusplus
}
#endif
#endif // GUMBO_UTIL_H_
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/vector.c
Vendored
+111
View File
@@ -0,0 +1,111 @@
/*
Copyright 2018 Craig Barnes.
Copyright 2010 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include "vector.h"
#include "util.h"
void gumbo_vector_init(unsigned int initial_capacity, GumboVector* vector) {
vector->length = 0;
vector->capacity = initial_capacity;
if (initial_capacity > 0) {
vector->data = gumbo_alloc(sizeof(void*) * initial_capacity);
} else {
vector->data = NULL;
}
}
void gumbo_vector_destroy(GumboVector* vector) {
if (vector->capacity > 0) {
gumbo_free(vector->data);
}
}
static void enlarge_vector_if_full(GumboVector* vector) {
if (vector->length >= vector->capacity) {
if (vector->capacity) {
vector->capacity *= 2;
size_t num_bytes = sizeof(void*) * vector->capacity;
vector->data = gumbo_realloc(vector->data, num_bytes);
} else {
// 0-capacity vector; no previous array to deallocate.
vector->capacity = 2;
vector->data = gumbo_alloc(sizeof(void*) * vector->capacity);
}
}
}
void gumbo_vector_add(void* element, GumboVector* vector) {
enlarge_vector_if_full(vector);
assert(vector->data);
assert(vector->length < vector->capacity);
vector->data[vector->length++] = element;
}
void* gumbo_vector_pop(GumboVector* vector) {
if (vector->length == 0) {
return NULL;
}
return vector->data[--vector->length];
}
int gumbo_vector_index_of(GumboVector* vector, const void* element) {
for (unsigned int i = 0; i < vector->length; ++i) {
if (vector->data[i] == element) {
return i;
}
}
return -1;
}
void gumbo_vector_insert_at (
void* element,
unsigned int index,
GumboVector* vector
) {
assert(index <= vector->length);
enlarge_vector_if_full(vector);
++vector->length;
memmove (
&vector->data[index + 1],
&vector->data[index],
sizeof(void*) * (vector->length - index - 1)
);
vector->data[index] = element;
}
void gumbo_vector_remove(void* node, GumboVector* vector) {
int index = gumbo_vector_index_of(vector, node);
if (index == -1) {
return;
}
gumbo_vector_remove_at(index, vector);
}
void* gumbo_vector_remove_at(unsigned int index, GumboVector* vector) {
assert(index < vector->length);
void* result = vector->data[index];
memmove (
&vector->data[index],
&vector->data[index + 1],
sizeof(void*) * (vector->length - index - 1)
);
--vector->length;
return result;
}
vendor/hachidori/third_party/hoshidicts/external/gumbo-parser/src/vector.h
Vendored
+45
View File
@@ -0,0 +1,45 @@
#ifndef GUMBO_VECTOR_H_
#define GUMBO_VECTOR_H_
#include "nokogiri_gumbo.h"
#ifdef __cplusplus
extern "C" {
#endif
// Initializes a new GumboVector with the specified initial capacity.
void gumbo_vector_init(unsigned int initial_capacity, GumboVector* vector);
// Frees the memory used by a GumboVector. Does not free the contained
// pointers.
void gumbo_vector_destroy(GumboVector* vector);
// Adds a new element to a GumboVector.
void gumbo_vector_add(void* element, GumboVector* vector);
// Removes and returns the element most recently added to the GumboVector.
// Ownership is transferred to caller. Capacity is unchanged. If the vector is
// empty, NULL is returned.
void* gumbo_vector_pop(GumboVector* vector);
// Inserts an element at a specific index. This is potentially O(N) time, but
// is necessary for some of the spec's behavior.
void gumbo_vector_insert_at (
void* element,
unsigned int index,
GumboVector* vector
);
// Removes an element from the vector, or does nothing if the element is not in
// the vector.
void gumbo_vector_remove(void* element, GumboVector* vector);
// Removes and returns an element at a specific index. Note that this is
// potentially O(N) time and should be used sparingly.
void* gumbo_vector_remove_at(unsigned int index, GumboVector* vector);
#ifdef __cplusplus
}
#endif
#endif // GUMBO_VECTOR_H_
vendor/hachidori/third_party/hoshidicts/external/lzokay/LICENSE
Vendored
+22
View File
@@ -0,0 +1,22 @@
The MIT License
Copyright (c) 2018 Jack Andersen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
vendor/hachidori/third_party/hoshidicts/external/lzokay/README.md
Vendored
+8
View File
@@ -0,0 +1,8 @@
# lzokay (vendored)
LZO1X decompressor/compressor used for MDX/MDD record blocks with compression
type 1.
Source: https://github.com/AxioDL/lzokay, commit
db2df1fcbebc2ed06c10f727f72567d40f06a2be (`lzokay.cpp`, `lzokay.hpp`,
`LICENSE`). Unmodified. Licence: MIT.
vendor/hachidori/third_party/hoshidicts/external/lzokay/lzokay.cpp
Vendored
+647
View File
@@ -0,0 +1,647 @@
#include "lzokay.hpp"
#include <cstring>
#include <algorithm>
#include <iterator>
/*
* Based on documentation from the Linux sources: Documentation/lzo.txt
* https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/lzo.txt
*/
namespace lzokay {
#if _WIN32
#define HOST_BIG_ENDIAN 0
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define HOST_BIG_ENDIAN 1
#else
#define HOST_BIG_ENDIAN 0
#endif
#if HOST_BIG_ENDIAN
static uint16_t get_le16(const uint8_t* p) {
uint16_t val = *reinterpret_cast<const uint16_t*>(p);
#if __GNUC__
return __builtin_bswap16(val);
#elif _WIN32
return _byteswap_ushort(val);
#else
return (val = (val << 8) | ((val >> 8) & 0xFF));
#endif
}
#else
static uint16_t get_le16(const uint8_t* p) {
return *reinterpret_cast<const uint16_t*>(p);
}
#endif
constexpr std::size_t Max255Count = std::size_t(~0) / 255 - 2;
#define NEEDS_IN(count) \
if (inp + (count) > inp_end) { \
dst_size = outp - dst; \
return EResult::InputOverrun; \
}
#define NEEDS_OUT(count) \
if (outp + (count) > outp_end) { \
dst_size = outp - dst; \
return EResult::OutputOverrun; \
}
#define CONSUME_ZERO_BYTE_LENGTH \
std::size_t offset; \
{ \
const uint8_t *old_inp = inp; \
while (*inp == 0) ++inp; \
offset = inp - old_inp; \
if (offset > Max255Count) { \
dst_size = outp - dst; \
return EResult::Error; \
} \
}
#define WRITE_ZERO_BYTE_LENGTH(length) \
{ \
std::size_t l; \
for (l = length; l > 255; l -= 255) { *outp++ = 0; } \
*outp++ = l; \
}
constexpr uint32_t M1MaxOffset = 0x0400;
constexpr uint32_t M2MaxOffset = 0x0800;
constexpr uint32_t M3MaxOffset = 0x4000;
constexpr uint32_t M4MaxOffset = 0xbfff;
constexpr uint32_t M1MinLen = 2;
constexpr uint32_t M1MaxLen = 2;
constexpr uint32_t M2MinLen = 3;
constexpr uint32_t M2MaxLen = 8;
constexpr uint32_t M3MinLen = 3;
constexpr uint32_t M3MaxLen = 33;
constexpr uint32_t M4MinLen = 3;
constexpr uint32_t M4MaxLen = 9;
constexpr uint32_t M1Marker = 0x0;
constexpr uint32_t M2Marker = 0x40;
constexpr uint32_t M3Marker = 0x20;
constexpr uint32_t M4Marker = 0x10;
constexpr uint32_t MaxMatchByLengthLen = 34; /* Max M3 len + 1 */
EResult decompress(const uint8_t* src, std::size_t src_size,
uint8_t* dst, std::size_t init_dst_size,
std::size_t& dst_size) {
dst_size = init_dst_size;
if (src_size < 3) {
dst_size = 0;
return EResult::InputOverrun;
}
const uint8_t* inp = src;
const uint8_t* inp_end = src + src_size;
uint8_t* outp = dst;
uint8_t* outp_end = dst + dst_size;
uint8_t* lbcur;
std::size_t lblen;
std::size_t state = 0;
std::size_t nstate = 0;
/* First byte encoding */
if (*inp >= 22) {
/* 22..255 : copy literal string
* length = (byte - 17) = 4..238
* state = 4 [ don't copy extra literals ]
* skip byte
*/
std::size_t len = *inp++ - uint8_t(17);
NEEDS_IN(len)
NEEDS_OUT(len)
for (std::size_t i = 0; i < len; ++i)
*outp++ = *inp++;
state = 4;
} else if (*inp >= 18) {
/* 18..21 : copy 0..3 literals
* state = (byte - 17) = 0..3 [ copy <state> literals ]
* skip byte
*/
nstate = *inp++ - uint8_t(17);
state = nstate;
NEEDS_IN(nstate)
NEEDS_OUT(nstate)
for (std::size_t i = 0; i < nstate; ++i)
*outp++ = *inp++;
}
/* 0..17 : follow regular instruction encoding, see below. It is worth
* noting that codes 16 and 17 will represent a block copy from
* the dictionary which is empty, and that they will always be
* invalid at this place.
*/
while (true) {
NEEDS_IN(1)
uint8_t inst = *inp++;
if (inst & 0xC0) {
/* [M2]
* 1 L L D D D S S (128..255)
* Copy 5-8 bytes from block within 2kB distance
* state = S (copy S literals after this block)
* length = 5 + L
* Always followed by exactly one byte : H H H H H H H H
* distance = (H << 3) + D + 1
*
* 0 1 L D D D S S (64..127)
* Copy 3-4 bytes from block within 2kB distance
* state = S (copy S literals after this block)
* length = 3 + L
* Always followed by exactly one byte : H H H H H H H H
* distance = (H << 3) + D + 1
*/
NEEDS_IN(1)
lbcur = outp - ((*inp++ << 3) + ((inst >> 2) & 0x7) + 1);
lblen = std::size_t(inst >> 5) + 1;
nstate = inst & uint8_t(0x3);
} else if (inst & M3Marker) {
/* [M3]
* 0 0 1 L L L L L (32..63)
* Copy of small block within 16kB distance (preferably less than 34B)
* length = 2 + (L ?: 31 + (zero_bytes * 255) + non_zero_byte)
* Always followed by exactly one LE16 : D D D D D D D D : D D D D D D S S
* distance = D + 1
* state = S (copy S literals after this block)
*/
lblen = std::size_t(inst & uint8_t(0x1f)) + 2;
if (lblen == 2) {
CONSUME_ZERO_BYTE_LENGTH
NEEDS_IN(1)
lblen += offset * 255 + 31 + *inp++;
}
NEEDS_IN(2)
nstate = get_le16(inp);
inp += 2;
lbcur = outp - ((nstate >> 2) + 1);
nstate &= 0x3;
} else if (inst & M4Marker) {
/* [M4]
* 0 0 0 1 H L L L (16..31)
* Copy of a block within 16..48kB distance (preferably less than 10B)
* length = 2 + (L ?: 7 + (zero_bytes * 255) + non_zero_byte)
* Always followed by exactly one LE16 : D D D D D D D D : D D D D D D S S
* distance = 16384 + (H << 14) + D
* state = S (copy S literals after this block)
* End of stream is reached if distance == 16384
*/
lblen = std::size_t(inst & uint8_t(0x7)) + 2;
if (lblen == 2) {
CONSUME_ZERO_BYTE_LENGTH
NEEDS_IN(1)
lblen += offset * 255 + 7 + *inp++;
}
NEEDS_IN(2)
nstate = get_le16(inp);
inp += 2;
lbcur = outp - (((inst & 0x8) << 11) + (nstate >> 2));
nstate &= 0x3;
if (lbcur == outp)
break; /* Stream finished */
lbcur -= 16384;
} else {
/* [M1] Depends on the number of literals copied by the last instruction. */
if (state == 0) {
/* If last instruction did not copy any literal (state == 0), this
* encoding will be a copy of 4 or more literal, and must be interpreted
* like this :
*
* 0 0 0 0 L L L L (0..15) : copy long literal string
* length = 3 + (L ?: 15 + (zero_bytes * 255) + non_zero_byte)
* state = 4 (no extra literals are copied)
*/
std::size_t len = inst + 3;
if (len == 3) {
CONSUME_ZERO_BYTE_LENGTH
NEEDS_IN(1)
len += offset * 255 + 15 + *inp++;
}
/* copy_literal_run */
NEEDS_IN(len)
NEEDS_OUT(len)
for (std::size_t i = 0; i < len; ++i)
*outp++ = *inp++;
state = 4;
continue;
} else if (state != 4) {
/* If last instruction used to copy between 1 to 3 literals (encoded in
* the instruction's opcode or distance), the instruction is a copy of a
* 2-byte block from the dictionary within a 1kB distance. It is worth
* noting that this instruction provides little savings since it uses 2
* bytes to encode a copy of 2 other bytes but it encodes the number of
* following literals for free. It must be interpreted like this :
*
* 0 0 0 0 D D S S (0..15) : copy 2 bytes from <= 1kB distance
* length = 2
* state = S (copy S literals after this block)
* Always followed by exactly one byte : H H H H H H H H
* distance = (H << 2) + D + 1
*/
NEEDS_IN(1)
nstate = inst & uint8_t(0x3);
lbcur = outp - ((inst >> 2) + (*inp++ << 2) + 1);
lblen = 2;
} else {
/* If last instruction used to copy 4 or more literals (as detected by
* state == 4), the instruction becomes a copy of a 3-byte block from the
* dictionary from a 2..3kB distance, and must be interpreted like this :
*
* 0 0 0 0 D D S S (0..15) : copy 3 bytes from 2..3 kB distance
* length = 3
* state = S (copy S literals after this block)
* Always followed by exactly one byte : H H H H H H H H
* distance = (H << 2) + D + 2049
*/
NEEDS_IN(1)
nstate = inst & uint8_t(0x3);
lbcur = outp - ((inst >> 2) + (*inp++ << 2) + 2049);
lblen = 3;
}
}
if (lbcur < dst) {
dst_size = outp - dst;
return EResult::LookbehindOverrun;
}
NEEDS_IN(nstate)
NEEDS_OUT(lblen + nstate)
/* Copy lookbehind */
for (std::size_t i = 0; i < lblen; ++i)
*outp++ = *lbcur++;
state = nstate;
/* Copy literal */
for (std::size_t i = 0; i < nstate; ++i)
*outp++ = *inp++;
}
dst_size = outp - dst;
if (lblen != 3) /* Ensure terminating M4 was encountered */
return EResult::Error;
if (inp == inp_end)
return EResult::Success;
else if (inp < inp_end)
return EResult::InputNotConsumed;
else
return EResult::InputOverrun;
}
struct State {
const uint8_t* src;
const uint8_t* src_end;
const uint8_t* inp;
uint32_t wind_sz;
uint32_t wind_b;
uint32_t wind_e;
uint32_t cycle1_countdown;
const uint8_t* bufp;
uint32_t buf_sz;
/* Access next input byte and advance both ends of circular buffer */
void get_byte(uint8_t* buf) {
if (inp >= src_end) {
if (wind_sz > 0)
--wind_sz;
buf[wind_e] = 0;
if (wind_e < DictBase::MaxMatchLen)
buf[DictBase::BufSize + wind_e] = 0;
} else {
buf[wind_e] = *inp;
if (wind_e < DictBase::MaxMatchLen)
buf[DictBase::BufSize + wind_e] = *inp;
++inp;
}
if (++wind_e == DictBase::BufSize)
wind_e = 0;
if (++wind_b == DictBase::BufSize)
wind_b = 0;
}
uint32_t pos2off(uint32_t pos) const {
return wind_b > pos ? wind_b - pos : DictBase::BufSize - (pos - wind_b);
}
};
class DictImpl : public DictBase {
public:
struct Match3Impl : DictBase::Match3 {
static uint32_t make_key(const uint8_t* data) {
return ((0x9f5f * (((uint32_t(data[0]) << 5 ^ uint32_t(data[1])) << 5) ^ data[2])) >> 5) & 0x3fff;
}
uint16_t get_head(uint32_t key) const {
return (chain_sz[key] == 0) ? uint16_t(UINT16_MAX) : head[key];
}
void init() {
std::fill(std::begin(chain_sz), std::end(chain_sz), 0);
}
void remove(uint32_t pos, const uint8_t* b) {
--chain_sz[make_key(b + pos)];
}
void advance(State& s, uint32_t& match_pos, uint32_t& match_count, const uint8_t* b) {
uint32_t key = make_key(b + s.wind_b);
match_pos = chain[s.wind_b] = get_head(key);
match_count = chain_sz[key]++;
if (match_count > DictBase::MaxMatchLen)
match_count = DictBase::MaxMatchLen;
head[key] = uint16_t(s.wind_b);
}
void skip_advance(State& s, const uint8_t* b) {
uint32_t key = make_key(b + s.wind_b);
chain[s.wind_b] = get_head(key);
head[key] = uint16_t(s.wind_b);
best_len[s.wind_b] = uint16_t(DictBase::MaxMatchLen + 1);
chain_sz[key]++;
}
};
struct Match2Impl : DictBase::Match2 {
static uint32_t make_key(const uint8_t* data) {
return uint32_t(data[0]) ^ (uint32_t(data[1]) << 8);
}
void init() {
std::fill(std::begin(head), std::end(head), UINT16_MAX);
}
void add(uint16_t pos, const uint8_t* b) {
head[make_key(b + pos)] = pos;
}
void remove(uint32_t pos, const uint8_t* b) {
uint16_t& p = head[make_key(b + pos)];
if (p == pos)
p = UINT16_MAX;
}
bool search(State& s, uint32_t& lb_pos, uint32_t& lb_len,
uint32_t best_pos[MaxMatchByLengthLen], const uint8_t* b) const {
uint16_t pos = head[make_key(b + s.wind_b)];
if (pos == UINT16_MAX)
return false;
if (best_pos[2] == 0)
best_pos[2] = pos + 1;
if (lb_len < 2) {
lb_len = 2;
lb_pos = pos;
}
return true;
}
};
void init(State& s, const uint8_t* src, std::size_t src_size) {
auto& match3 = static_cast<Match3Impl&>(_storage->match3);
auto& match2 = static_cast<Match2Impl&>(_storage->match2);
s.cycle1_countdown = DictBase::MaxDist;
match3.init();
match2.init();
s.src = src;
s.src_end = src + src_size;
s.inp = src;
s.wind_sz = uint32_t(std::min(src_size, std::size_t(MaxMatchLen)));
s.wind_b = 0;
s.wind_e = s.wind_sz;
std::copy_n(s.inp, s.wind_sz, _storage->buffer);
s.inp += s.wind_sz;
if (s.wind_e == DictBase::BufSize)
s.wind_e = 0;
if (s.wind_sz < 3)
std::fill_n(_storage->buffer + s.wind_b + s.wind_sz, 3, 0);
}
void reset_next_input_entry(State& s, Match3Impl& match3, Match2Impl& match2) {
/* Remove match from about-to-be-clobbered buffer entry */
if (s.cycle1_countdown == 0) {
match3.remove(s.wind_e, _storage->buffer);
match2.remove(s.wind_e, _storage->buffer);
} else {
--s.cycle1_countdown;
}
}
void advance(State& s, uint32_t& lb_off, uint32_t& lb_len,
uint32_t best_off[MaxMatchByLengthLen], bool skip) {
auto& match3 = static_cast<Match3Impl&>(_storage->match3);
auto& match2 = static_cast<Match2Impl&>(_storage->match2);
if (skip) {
for (uint32_t i = 0; i < lb_len - 1; ++i) {
reset_next_input_entry(s, match3, match2);
match3.skip_advance(s, _storage->buffer);
match2.add(uint16_t(s.wind_b), _storage->buffer);
s.get_byte(_storage->buffer);
}
}
lb_len = 1;
lb_off = 0;
uint32_t lb_pos;
uint32_t best_pos[MaxMatchByLengthLen] = {};
uint32_t match_pos, match_count;
match3.advance(s, match_pos, match_count, _storage->buffer);
int best_char = _storage->buffer[s.wind_b];
uint32_t best_len = lb_len;
if (lb_len >= s.wind_sz) {
if (s.wind_sz == 0)
best_char = -1;
lb_off = 0;
match3.best_len[s.wind_b] = DictBase::MaxMatchLen + 1;
} else {
if (match2.search(s, lb_pos, lb_len, best_pos, _storage->buffer) && s.wind_sz >= 3) {
for (uint32_t i = 0; i < match_count; ++i, match_pos = match3.chain[match_pos]) {
auto ref_ptr = _storage->buffer + s.wind_b;
auto match_ptr = _storage->buffer + match_pos;
auto mismatch = std::mismatch(ref_ptr, ref_ptr + s.wind_sz, match_ptr);
auto match_len = uint32_t(mismatch.first - ref_ptr);
if (match_len < 2)
continue;
if (match_len < MaxMatchByLengthLen && best_pos[match_len] == 0)
best_pos[match_len] = match_pos + 1;
if (match_len > lb_len) {
lb_len = match_len;
lb_pos = match_pos;
if (match_len == s.wind_sz || match_len > match3.best_len[match_pos])
break;
}
}
}
if (lb_len > best_len)
lb_off = s.pos2off(lb_pos);
match3.best_len[s.wind_b] = uint16_t(lb_len);
for (auto posit = std::begin(best_pos) + 2, offit = best_off + 2;
posit != std::end(best_pos); ++posit, ++offit) {
*offit = (*posit > 0) ? s.pos2off(*posit - 1) : 0;
}
}
reset_next_input_entry(s, match3, match2);
match2.add(uint16_t(s.wind_b), _storage->buffer);
s.get_byte(_storage->buffer);
if (best_char < 0) {
s.buf_sz = 0;
lb_len = 0;
/* Signal exit */
} else {
s.buf_sz = s.wind_sz + 1;
}
s.bufp = s.inp - s.buf_sz;
}
};
static void find_better_match(const uint32_t best_off[MaxMatchByLengthLen], uint32_t& lb_len, uint32_t& lb_off) {
if (lb_len <= M2MinLen || lb_off <= M2MaxOffset)
return;
if (lb_off > M2MaxOffset && lb_len >= M2MinLen + 1 && lb_len <= M2MaxLen + 1 &&
best_off[lb_len - 1] != 0 && best_off[lb_len - 1] <= M2MaxOffset) {
lb_len -= 1;
lb_off = best_off[lb_len];
} else if (lb_off > M3MaxOffset && lb_len >= M4MaxLen + 1 && lb_len <= M2MaxLen + 2 &&
best_off[lb_len - 2] && best_off[lb_len] <= M2MaxOffset) {
lb_len -= 2;
lb_off = best_off[lb_len];
} else if (lb_off > M3MaxOffset && lb_len >= M4MaxLen + 1 && lb_len <= M3MaxLen + 1 &&
best_off[lb_len - 1] != 0 && best_off[lb_len - 2] <= M3MaxOffset) {
lb_len -= 1;
lb_off = best_off[lb_len];
}
}
static EResult encode_literal_run(uint8_t*& outp, const uint8_t* outp_end, const uint8_t* dst, std::size_t& dst_size,
const uint8_t* lit_ptr, uint32_t lit_len) {
if (outp == dst && lit_len <= 238) {
NEEDS_OUT(1);
*outp++ = uint8_t(17 + lit_len);
} else if (lit_len <= 3) {
outp[-2] = uint8_t(outp[-2] | lit_len);
} else if (lit_len <= 18) {
NEEDS_OUT(1);
*outp++ = uint8_t(lit_len - 3);
} else {
NEEDS_OUT((lit_len - 18) / 255 + 2);
*outp++ = 0;
WRITE_ZERO_BYTE_LENGTH(lit_len - 18);
}
NEEDS_OUT(lit_len);
outp = std::copy_n(lit_ptr, lit_len, outp);
return EResult::Success;
}
static EResult encode_lookback_match(uint8_t*& outp, const uint8_t* outp_end, const uint8_t* dst, std::size_t& dst_size,
uint32_t lb_len, uint32_t lb_off, uint32_t last_lit_len) {
if (lb_len == 2) {
lb_off -= 1;
NEEDS_OUT(2);
*outp++ = uint8_t(M1Marker | ((lb_off & 0x3) << 2));
*outp++ = uint8_t(lb_off >> 2);
} else if (lb_len <= M2MaxLen && lb_off <= M2MaxOffset) {
lb_off -= 1;
NEEDS_OUT(2);
*outp++ = uint8_t((lb_len - 1) << 5 | ((lb_off & 0x7) << 2));
*outp++ = uint8_t(lb_off >> 3);
} else if (lb_len == M2MinLen && lb_off <= M1MaxOffset + M2MaxOffset && last_lit_len >= 4) {
lb_off -= 1 + M2MaxOffset;
NEEDS_OUT(2);
*outp++ = uint8_t(M1Marker | ((lb_off & 0x3) << 2));
*outp++ = uint8_t(lb_off >> 2);
} else if (lb_off <= M3MaxOffset) {
lb_off -= 1;
if (lb_len <= M3MaxLen) {
NEEDS_OUT(1);
*outp++ = uint8_t(M3Marker | (lb_len - 2));
} else {
lb_len -= M3MaxLen;
NEEDS_OUT(lb_len / 255 + 2);
*outp++ = uint8_t(M3Marker);
WRITE_ZERO_BYTE_LENGTH(lb_len);
}
NEEDS_OUT(2);
*outp++ = uint8_t(lb_off << 2);
*outp++ = uint8_t(lb_off >> 6);
} else {
lb_off -= 0x4000;
if (lb_len <= M4MaxLen) {
NEEDS_OUT(1);
*outp++ = uint8_t(M4Marker | ((lb_off & 0x4000) >> 11) | (lb_len - 2));
} else {
lb_len -= M4MaxLen;
NEEDS_OUT(lb_len / 255 + 2);
*outp++ = uint8_t(M4Marker | ((lb_off & 0x4000) >> 11));
WRITE_ZERO_BYTE_LENGTH(lb_len);
}
NEEDS_OUT(2);
*outp++ = uint8_t(lb_off << 2);
*outp++ = uint8_t(lb_off >> 6);
}
return EResult::Success;
}
EResult compress(const uint8_t* src, std::size_t src_size,
uint8_t* dst, std::size_t init_dst_size,
std::size_t& dst_size, DictBase& dict) {
EResult err;
State s;
auto& d = static_cast<DictImpl&>(dict);
dst_size = init_dst_size;
uint8_t* outp = dst;
uint8_t* outp_end = dst + dst_size;
uint32_t lit_len = 0;
uint32_t lb_off, lb_len;
uint32_t best_off[MaxMatchByLengthLen];
d.init(s, src, src_size);
const uint8_t* lit_ptr = s.inp;
d.advance(s, lb_off, lb_len, best_off, false);
while (s.buf_sz > 0) {
if (lit_len == 0)
lit_ptr = s.bufp;
if (lb_len < 2 || (lb_len == 2 && (lb_off > M1MaxOffset || lit_len == 0 || lit_len >= 4)) ||
(lb_len == 2 && outp == dst) || (outp == dst && lit_len == 0)) {
lb_len = 0;
} else if (lb_len == M2MinLen && lb_off > M1MaxOffset + M2MaxOffset && lit_len >= 4) {
lb_len = 0;
}
if (lb_len == 0) {
++lit_len;
d.advance(s, lb_off, lb_len, best_off, false);
continue;
}
find_better_match(best_off, lb_len, lb_off);
if ((err = encode_literal_run(outp, outp_end, dst, dst_size, lit_ptr, lit_len)) < EResult::Success)
return err;
if ((err = encode_lookback_match(outp, outp_end, dst, dst_size, lb_len, lb_off, lit_len)) < EResult::Success)
return err;
lit_len = 0;
d.advance(s, lb_off, lb_len, best_off, true);
}
if ((err = encode_literal_run(outp, outp_end, dst, dst_size, lit_ptr, lit_len)) < EResult::Success)
return err;
/* Terminating M4 */
NEEDS_OUT(3);
*outp++ = M4Marker | 1;
*outp++ = 0;
*outp++ = 0;
dst_size = outp - dst;
return EResult::Success;
}
}
vendor/hachidori/third_party/hoshidicts/external/lzokay/lzokay.hpp
Vendored
+79
View File
@@ -0,0 +1,79 @@
#pragma once
#include <cstddef>
#include <cstdint>
#include <memory>
namespace lzokay {
enum class EResult {
LookbehindOverrun = -4,
OutputOverrun = -3,
InputOverrun = -2,
Error = -1,
Success = 0,
InputNotConsumed = 1,
};
class DictBase {
protected:
static constexpr uint32_t HashSize = 0x4000;
static constexpr uint32_t MaxDist = 0xbfff;
static constexpr uint32_t MaxMatchLen = 0x800;
static constexpr uint32_t BufSize = MaxDist + MaxMatchLen;
/* List encoding of previous 3-byte data matches */
struct Match3 {
uint16_t head[HashSize]; /* key -> chain-head-pos */
uint16_t chain_sz[HashSize]; /* key -> chain-size */
uint16_t chain[BufSize]; /* chain-pos -> next-chain-pos */
uint16_t best_len[BufSize]; /* chain-pos -> best-match-length */
};
/* Encoding of 2-byte data matches */
struct Match2 {
uint16_t head[1 << 16]; /* 2-byte-data -> head-pos */
};
struct Data {
Match3 match3;
Match2 match2;
/* Circular buffer caching enough data to access the maximum lookback
* distance of 48K + maximum match length of 2K. An additional 2K is
* allocated so the start of the buffer may be replicated at the end,
* therefore providing efficient circular access.
*/
uint8_t buffer[BufSize + MaxMatchLen];
};
using storage_type = Data;
storage_type* _storage;
DictBase() = default;
friend struct State;
friend EResult compress(const uint8_t* src, std::size_t src_size,
uint8_t* dst, std::size_t& dst_size, DictBase& dict);
};
template <template<typename> class _Alloc = std::allocator>
class Dict : public DictBase {
_Alloc<DictBase::storage_type> _allocator;
public:
Dict() { _storage = _allocator.allocate(1); }
~Dict() { _allocator.deallocate(_storage, 1); }
};
EResult decompress(const uint8_t* src, std::size_t src_size,
uint8_t* dst, std::size_t dst_size,
std::size_t& out_size);
EResult compress(const uint8_t* src, std::size_t src_size,
uint8_t* dst, std::size_t dst_size,
std::size_t& out_size, DictBase& dict);
inline EResult compress(const uint8_t* src, std::size_t src_size,
uint8_t* dst, std::size_t dst_size,
std::size_t& out_size) {
Dict<> dict;
return compress(src, src_size, dst, dst_size, out_size, dict);
}
constexpr std::size_t compress_worst_size(std::size_t s) {
return s + s / 16 + 64 + 3;
}
}