[modules] Add support for 'textual header' directives.

This allows a module to specify that it logically contains a file, but that
said file is non-modular and intended for textual inclusion. This allows
layering checks to work properly in the presence of such files.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@220448 91177308-0d34-0410-b5e6-96231b3b80d8

docs/Modules.rst

@@ -267,6 +267,12 @@ As an example, the module map file for the C standard library might look a bit l
 .. parsed-literal::
 
   module std [system] [extern_c] {
+    module assert {
+      textual header "assert.h"
+      header "bits/assert-decls.h"
+      export *
+    }
+
     module complex {
       header "complex.h"
       export *
@@ -299,11 +305,11 @@ Module map files use a simplified form of the C99 lexer, with the same rules for
 
 .. parsed-literal::
 
-  ``config_macros`` ``export``     ``module``
+  ``config_macros`` ``export``     ``private``
   ``conflict``      ``framework``  ``requires``
-  ``exclude``       ``header``     ``private``
+  ``exclude``       ``header``     ``textual``
   ``explicit``      ``link``       ``umbrella``
-  ``extern``        ``use``
+  ``extern``        ``module``     ``use``
 
 Module map file
 ---------------
@@ -331,7 +337,7 @@ A module declaration describes a module, including the headers that contribute t
     ``explicit``:sub:`opt` ``framework``:sub:`opt` ``module`` *module-id* *attributes*:sub:`opt` '{' *module-member** '}'
     ``extern`` ``module`` *module-id* *string-literal*
 
-The *module-id* should consist of only a single *identifier*, which provides the name of the module being defined. Each module shall have a single definition. 
+The *module-id* should consist of only a single *identifier*, which provides the name of the module being defined. Each module shall have a single definition.
 
 The ``explicit`` qualifier can only be applied to a submodule, i.e., a module that is nested within another module. The contents of explicit submodules are only made available when the submodule itself was explicitly named in an import declaration or was re-exported from an imported module.
 
@@ -441,9 +447,10 @@ A header declaration specifies that a particular header is associated with the e
   *header-declaration*:
     ``umbrella``:sub:`opt` ``header`` *string-literal*
     ``private`` ``header`` *string-literal*
+    ``textual`` ``header`` *string-literal*
     ``exclude`` ``header`` *string-literal*
 
-A header declaration that does not contain ``exclude`` specifies a header that contributes to the enclosing module. Specifically, when the module is built, the named header will be parsed and its declarations will be (logically) placed into the enclosing submodule.
+A header declaration that does not contain ``exclude`` nor ``textual`` specifies a header that contributes to the enclosing module. Specifically, when the module is built, the named header will be parsed and its declarations will be (logically) placed into the enclosing submodule.
 
 A header with the ``umbrella`` specifier is called an umbrella header. An umbrella header includes all of the headers within its directory (and any subdirectories), and is typically used (in the ``#include`` world) to easily access the full API provided by a particular library. With modules, an umbrella header is a convenient shortcut that eliminates the need to write out ``header`` declarations for every library header. A given directory can only contain a single umbrella header.
 
@@ -455,6 +462,8 @@ A header with the ``umbrella`` specifier is called an umbrella header. An umbrel
 
 A header with the ``private`` specifier may not be included from outside the module itself.
 
+A header with the ``textual`` specifier will not be included when the module is built, and will be textually included if it is named by a ``#include`` directive. However, it is considered to be part of the module for the purpose of checking *use-declaration*\s.
+
 A header with the ``exclude`` specifier is excluded from the module. It will not be included when the module is built, nor will it be considered to be part of the module.
 
 **Example**: The C header ``assert.h`` is an excellent candidate for an excluded header, because it is meant to be included multiple times (possibly with different ``NDEBUG`` settings).

include/clang/Basic/Module.h

@@ -87,6 +87,10 @@ public:
   /// \brief The headers that are explicitly excluded from this module.
   SmallVector<const FileEntry *, 2> ExcludedHeaders;
 
+  /// \brief The headers that are logically part of this module but
+  /// must be textually included.
+  SmallVector<const FileEntry *, 2> TextualHeaders;
+
   /// \brief The headers that are private to this module.
   SmallVector<const FileEntry *, 2> PrivateHeaders;
 

include/clang/Lex/ModuleMap.h

@@ -71,6 +71,9 @@ public:
     NormalHeader,
     /// \brief This header is included but private.
     PrivateHeader,
+    /// \brief This header is part of the module (for layering purposes) but
+    /// should be textually included.
+    TextualHeader,
     /// \brief This header is explicitly excluded from the module.
     ExcludedHeader
     // Caution: Adding an enumerator needs other changes.
@@ -249,11 +252,16 @@ public:
   /// used from.  Used to disambiguate if a header is present in multiple
   /// modules.
   ///
+  /// \param IncludeTextualHeaders If \c true, also find textual headers. By
+  /// default, these are treated like excluded headers and result in no known
+  /// header being found.
+  ///
   /// \returns The module KnownHeader, which provides the module that owns the
   /// given header file.  The KnownHeader is default constructed to indicate
   /// that no module owns this header file.
   KnownHeader findModuleForHeader(const FileEntry *File,
-                                  Module *RequestingModule = nullptr);
+                                  Module *RequestingModule = nullptr,
+                                  bool IncludeTextualHeaders = false);
 
   /// \brief Reports errors if a module must not include a specific file.
   ///

include/clang/Serialization/ASTBitCodes.h

@@ -638,7 +638,10 @@ namespace clang {
       /// \brief Specifies a conflict with another module.
       SUBMODULE_CONFLICT = 12,
       /// \brief Specifies a header that is private to this submodule.
-      SUBMODULE_PRIVATE_HEADER = 13
+      SUBMODULE_PRIVATE_HEADER = 13,
+      /// \brief Specifies a header that is part of the module but must be
+      /// textually included.
+      SUBMODULE_TEXTUAL_HEADER = 14,
     };
 
     /// \brief Record types used within a comments block.

lib/Basic/Module.cpp

@@ -338,27 +338,23 @@ void Module::print(raw_ostream &OS, unsigned Indent) const {
     OS << "\n";
   }
 
-  for (unsigned I = 0, N = NormalHeaders.size(); I != N; ++I) {
-    OS.indent(Indent + 2);
-    OS << "header \"";
-    OS.write_escaped(NormalHeaders[I]->getName());
-    OS << "\"\n";
-  }
-
-  for (unsigned I = 0, N = ExcludedHeaders.size(); I != N; ++I) {
-    OS.indent(Indent + 2);
-    OS << "exclude header \"";
-    OS.write_escaped(ExcludedHeaders[I]->getName());
-    OS << "\"\n";
+  struct HeaderKind {
+    StringRef Prefix;
+    const SmallVectorImpl<const FileEntry *> &Headers;
+  } Kinds[] = {{"", NormalHeaders},
+               {"exclude ", ExcludedHeaders},
+               {"textual ", TextualHeaders},
+               {"private ", PrivateHeaders}};
+
+  for (auto &K : Kinds) {
+    for (auto *H : K.Headers) {
+      OS.indent(Indent + 2);
+      OS << K.Prefix << "header \"";
+      OS.write_escaped(H->getName());
+      OS << "\"\n";
+    }
   }
 
-  for (unsigned I = 0, N = PrivateHeaders.size(); I != N; ++I) {
-    OS.indent(Indent + 2);
-    OS << "private header \"";
-    OS.write_escaped(PrivateHeaders[I]->getName());
-    OS << "\"\n";
-  }
-  
   for (submodule_const_iterator MI = submodule_begin(), MIEnd = submodule_end();
        MI != MIEnd; ++MI)
     // Print inferred subframework modules so that we don't need to re-infer

lib/Lex/ModuleMap.cpp

@@ -315,9 +315,16 @@ void ModuleMap::diagnoseHeaderInclusion(Module *RequestingModule,
 
 ModuleMap::KnownHeader
 ModuleMap::findModuleForHeader(const FileEntry *File,
-                               Module *RequestingModule) {
+                               Module *RequestingModule,
+                               bool IncludeTextualHeaders) {
   HeadersMap::iterator Known = findKnownHeader(File);
 
+  auto MakeResult = [&](ModuleMap::KnownHeader R) -> ModuleMap::KnownHeader {
+    if (!IncludeTextualHeaders && R.getRole() == ModuleMap::TextualHeader)
+      return ModuleMap::KnownHeader();
+    return R;
+  };
+
   if (Known != Headers.end()) {
     ModuleMap::KnownHeader Result = KnownHeader();
 
@@ -336,7 +343,7 @@ ModuleMap::findModuleForHeader(const FileEntry *File,
       // If 'File' is part of 'RequestingModule', 'RequestingModule' is the
       // module we are looking for.
       if (I->getModule() == RequestingModule)
-        return *I;
+        return MakeResult(*I);
 
       // If uses need to be specified explicitly, we are only allowed to return
       // modules that are explicitly used by the requesting module.
@@ -349,10 +356,11 @@ ModuleMap::findModuleForHeader(const FileEntry *File,
       // are going to get.
       // FIXME: If we have a RequestingModule, we should prefer the header from
       // that module.
-      if (I->getRole() == ModuleMap::NormalHeader)
+      if (I->getRole() == ModuleMap::NormalHeader ||
+          I->getRole() == ModuleMap::TextualHeader)
         break;
     }
-    return Result;
+    return MakeResult(Result);
   }
 
   SmallVector<const DirectoryEntry *, 2> SkippedDirs;
@@ -422,9 +430,9 @@ ModuleMap::findModuleForHeader(const FileEntry *File,
     if (!Result->isAvailable())
       return KnownHeader();
 
-    return Headers[File].back();
+    return MakeResult(Headers[File].back());
   }
-  
+
   return KnownHeader();
 }
 
@@ -785,6 +793,8 @@ void ModuleMap::addHeader(Module *Mod, const FileEntry *Header,
                           ModuleHeaderRole Role) {
   if (Role == ExcludedHeader) {
     Mod->ExcludedHeaders.push_back(Header);
+  } else if (Role == TextualHeader) {
+    Mod->TextualHeaders.push_back(Header);
   } else {
     if (Role == PrivateHeader)
       Mod->PrivateHeaders.push_back(Header);
@@ -946,6 +956,7 @@ namespace clang {
       RequiresKeyword,
       Star,
       StringLiteral,
+      TextualKeyword,
       LBrace,
       RBrace,
       LSquare,
@@ -1096,6 +1107,7 @@ retry:
                  .Case("module", MMToken::ModuleKeyword)
                  .Case("private", MMToken::PrivateKeyword)
                  .Case("requires", MMToken::RequiresKeyword)
+                 .Case("textual", MMToken::TextualKeyword)
                  .Case("umbrella", MMToken::UmbrellaKeyword)
                  .Case("use", MMToken::UseKeyword)
                  .Default(MMToken::Identifier);
@@ -1463,6 +1475,17 @@ void ModuleMapParser::parseModuleDecl() {
       parseRequiresDecl();
       break;
 
+    case MMToken::TextualKeyword: {
+      SourceLocation TextualLoc = consumeToken();
+      if (Tok.is(MMToken::HeaderKeyword)) {
+        parseHeaderDecl(MMToken::TextualKeyword, TextualLoc);
+      } else {
+        Diags.Report(Tok.getLocation(), diag::err_mmap_expected_header)
+          << "textual";
+      }
+      break;
+    }
+
     case MMToken::UmbrellaKeyword: {
       SourceLocation UmbrellaLoc = consumeToken();
       if (Tok.is(MMToken::HeaderKeyword))
@@ -1650,8 +1673,12 @@ static void appendSubframeworkPaths(Module *Mod,
 /// \brief Parse a header declaration.
 ///
 ///   header-declaration:
-///     'umbrella'[opt] 'header' string-literal
 ///     'exclude'[opt] 'header' string-literal
+///     'private'[opt] 'header' string-literal
+///     'textual'[opt] 'header' string-literal
+///     'umbrella'[opt] 'header' string-literal
+///
+/// FIXME: Support 'private textual header'.
 void ModuleMapParser::parseHeaderDecl(MMToken::TokenKind LeadingToken,
                                       SourceLocation LeadingLoc) {
   assert(Tok.is(MMToken::HeaderKeyword));
@@ -1747,6 +1774,8 @@ void ModuleMapParser::parseHeaderDecl(MMToken::TokenKind LeadingToken,
         Role = ModuleMap::ExcludedHeader;
       else if (LeadingToken == MMToken::PrivateKeyword)
         Role = ModuleMap::PrivateHeader;
+      else if (LeadingToken == MMToken::TextualKeyword)
+        Role = ModuleMap::TextualHeader;
       else
         assert(LeadingToken == MMToken::HeaderKeyword);
 
@@ -1839,6 +1868,7 @@ void ModuleMapParser::parseExportDecl() {
   ModuleId ParsedModuleId;
   bool Wildcard = false;
   do {
+    // FIXME: Support string-literal module names here.
     if (Tok.is(MMToken::Identifier)) {
       ParsedModuleId.push_back(std::make_pair(Tok.getString(), 
                                               Tok.getLocation()));
@@ -1936,6 +1966,7 @@ void ModuleMapParser::parseConfigMacros() {
   }
 
   // If we don't have an identifier, we're done.
+  // FIXME: Support macros with the same name as a keyword here.
   if (!Tok.is(MMToken::Identifier))
     return;
 
@@ -1952,6 +1983,7 @@ void ModuleMapParser::parseConfigMacros() {
     consumeToken();
 
     // We expect to see a macro name here.
+    // FIXME: Support macros with the same name as a keyword here.
     if (!Tok.is(MMToken::Identifier)) {
       Diags.Report(Tok.getLocation(), diag::err_mmap_expected_config_macro);
       break;
@@ -2117,6 +2149,7 @@ void ModuleMapParser::parseInferredModuleDecl(bool Framework, bool Explicit) {
       }
 
       consumeToken();
+      // FIXME: Support string-literal module names here.
       if (!Tok.is(MMToken::Identifier)) {
         Diags.Report(Tok.getLocation(), diag::err_mmap_missing_exclude_name);
         break;

lib/Lex/PPLexerChange.cpp

@@ -519,7 +519,8 @@ bool Preprocessor::HandleEndOfFile(Token &Result, bool isEndOfMacro) {
           continue;
 
         // If it's not part of a module and not unknown, complain.
-        if (!ModMap.findModuleForHeader(File) &&
+        if (!ModMap.findModuleForHeader(File, nullptr,
+                                        /*IncludeTextualHeaders*/true) &&
             !ModMap.isHeaderInUnavailableModule(File)) {
           Diag(StartLoc, diag::warn_forgotten_module_header)
             << File->getName() << Mod->getFullModuleName();

lib/Serialization/ASTWriter.cpp

@@ -2404,6 +2404,11 @@ void ASTWriter::WriteSubmodules(Module *WritingModule) {
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
   unsigned ExcludedHeaderAbbrev = Stream.EmitAbbrev(Abbrev);
 
+  Abbrev = new BitCodeAbbrev();
+  Abbrev->Add(BitCodeAbbrevOp(SUBMODULE_TEXTUAL_HEADER));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
+  unsigned TextualHeaderAbbrev = Stream.EmitAbbrev(Abbrev);
+
   Abbrev = new BitCodeAbbrev();
   Abbrev->Add(BitCodeAbbrevOp(SUBMODULE_PRIVATE_HEADER));
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
@@ -2481,7 +2486,7 @@ void ASTWriter::WriteSubmodules(Module *WritingModule) {
       Stream.EmitRecordWithBlob(UmbrellaDirAbbrev, Record, 
                                 UmbrellaDir->getName());      
     }
-    
+
     // Emit the headers.
     for (unsigned I = 0, N = Mod->NormalHeaders.size(); I != N; ++I) {
       Record.clear();
@@ -2496,6 +2501,13 @@ void ASTWriter::WriteSubmodules(Module *WritingModule) {
       Stream.EmitRecordWithBlob(ExcludedHeaderAbbrev, Record, 
                                 Mod->ExcludedHeaders[I]->getName());
     }
+    // Emit the textual headers.
+    for (unsigned I = 0, N = Mod->TextualHeaders.size(); I != N; ++I) {
+      Record.clear();
+      Record.push_back(SUBMODULE_TEXTUAL_HEADER);
+      Stream.EmitRecordWithBlob(TextualHeaderAbbrev, Record, 
+                                Mod->TextualHeaders[I]->getName());
+    }
     // Emit the private headers.
     for (unsigned I = 0, N = Mod->PrivateHeaders.size(); I != N; ++I) {
       Record.clear();

test/Modules/Inputs/declare-use/module.map

@@ -38,6 +38,7 @@ module XG {
   use XC
   use XE
   use XJ
+  use XK
 }
 
 module XH {
@@ -52,5 +53,13 @@ module XJ {
   header "j.h"
 }
 
+module XK {
+  textual header "k.h"
+}
+
+module XL {
+  textual header "l.h"
+}
+
 module XS {
 }