[SVE][IR] Scalable Vector IR Type

* Adds a 'scalable' flag to VectorType
* Adds an 'ElementCount' class to VectorType to pass (possibly scalable) vector lengths, with overloaded operators.
* Modifies existing helper functions to use ElementCount
* Adds support for serializing/deserializing to/from both textual and bitcode IR formats
* Extends the verifier to reject global variables of scalable types
* Updates documentation

See the latest version of the RFC here: http://lists.llvm.org/pipermail/llvm-dev/2018-July/124396.html

Reviewers: rengolin, lattner, echristo, chandlerc, hfinkel, rkruppe, samparker, SjoerdMeijer, greened, sebpop

Reviewed By: hfinkel, sebpop

Differential Revision: https://reviews.llvm.org/D32530


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@361953 91177308-0d34-0410-b5e6-96231b3b80d8

docs/LangRef.rst

@@ -675,6 +675,9 @@ an optional list of attached :ref:`metadata <metadata>`.
 Variables and aliases can have a
 :ref:`Thread Local Storage Model <tls_model>`.
 
+:ref:`Scalable vectors <t_vector>` cannot be global variables or members of
+structs or arrays because their size is unknown at compile time.
+
 Syntax::
 
       @<GlobalVarName> = [Linkage] [PreemptionSpecifier] [Visibility]
@@ -2730,30 +2733,40 @@ Vector Type
 A vector type is a simple derived type that represents a vector of
 elements. Vector types are used when multiple primitive data are
 operated in parallel using a single instruction (SIMD). A vector type
-requires a size (number of elements) and an underlying primitive data
-type. Vector types are considered :ref:`first class <t_firstclass>`.
+requires a size (number of elements), an underlying primitive data type,
+and a scalable property to represent vectors where the exact hardware
+vector length is unknown at compile time. Vector types are considered
+:ref:`first class <t_firstclass>`.
 
 :Syntax:
 
 ::
 
-      < <# elements> x <elementtype> >
+      < <# elements> x <elementtype> >          ; Fixed-length vector
+      < vscale x <# elements> x <elementtype> > ; Scalable vector
 
 The number of elements is a constant integer value larger than 0;
 elementtype may be any integer, floating-point or pointer type. Vectors
-of size zero are not allowed.
+of size zero are not allowed. For scalable vectors, the total number of
+elements is a constant multiple (called vscale) of the specified number
+of elements; vscale is a positive integer that is unknown at compile time
+and the same hardware-dependent constant for all scalable vectors at run
+time. The size of a specific scalable vector type is thus constant within
+IR, even if the exact size in bytes cannot be determined until run time.
 
 :Examples:
 
-+-------------------+--------------------------------------------------+
-| ``<4 x i32>``     | Vector of 4 32-bit integer values.               |
-+-------------------+--------------------------------------------------+
-| ``<8 x float>``   | Vector of 8 32-bit floating-point values.        |
-+-------------------+--------------------------------------------------+
-| ``<2 x i64>``     | Vector of 2 64-bit integer values.               |
-+-------------------+--------------------------------------------------+
-| ``<4 x i64*>``    | Vector of 4 pointers to 64-bit integer values.   |
-+-------------------+--------------------------------------------------+
++------------------------+----------------------------------------------------+
+| ``<4 x i32>``          | Vector of 4 32-bit integer values.                 |
++------------------------+----------------------------------------------------+
+| ``<8 x float>``        | Vector of 8 32-bit floating-point values.          |
++------------------------+----------------------------------------------------+
+| ``<2 x i64>``          | Vector of 2 64-bit integer values.                 |
++------------------------+----------------------------------------------------+
+| ``<4 x i64*>``         | Vector of 4 pointers to 64-bit integer values.     |
++------------------------+----------------------------------------------------+
+| ``<vscale x 4 x i32>`` | Vector with a multiple of 4 32-bit integer values. |
++------------------------+----------------------------------------------------+
 
 .. _t_label:
 
@@ -8135,6 +8148,7 @@ Syntax:
 ::
 
       <result> = extractelement <n x <ty>> <val>, <ty2> <idx>  ; yields <ty>
+      <result> = extractelement <vscale x n x <ty>> <val>, <ty2> <idx> ; yields <ty>
 
 Overview:
 """""""""
@@ -8155,7 +8169,9 @@ Semantics:
 
 The result is a scalar of the same type as the element type of ``val``.
 Its value is the value at position ``idx`` of ``val``. If ``idx``
-exceeds the length of ``val``, the result is a
+exceeds the length of ``val`` for a fixed-length vector, the result is a
+:ref:`poison value <poisonvalues>`. For a scalable vector, if the value
+of ``idx`` exceeds the runtime length of the vector, the result is a
 :ref:`poison value <poisonvalues>`.
 
 Example:
@@ -8176,6 +8192,7 @@ Syntax:
 ::
 
       <result> = insertelement <n x <ty>> <val>, <ty> <elt>, <ty2> <idx>    ; yields <n x <ty>>
+      <result> = insertelement <vscale x n x <ty>> <val>, <ty> <elt>, <ty2> <idx> ; yields <vscale x n x <ty>>
 
 Overview:
 """""""""
@@ -8197,7 +8214,9 @@ Semantics:
 
 The result is a vector of the same type as ``val``. Its element values
 are those of ``val`` except at position ``idx``, where it gets the value
-``elt``. If ``idx`` exceeds the length of ``val``, the result
+``elt``. If ``idx`` exceeds the length of ``val`` for a fixed-length vector,
+the result is a :ref:`poison value <poisonvalues>`. For a scalable vector,
+if the value of ``idx`` exceeds the runtime length of the vector, the result
 is a :ref:`poison value <poisonvalues>`.
 
 Example:
@@ -8218,6 +8237,7 @@ Syntax:
 ::
 
       <result> = shufflevector <n x <ty>> <v1>, <n x <ty>> <v2>, <m x i32> <mask>    ; yields <m x <ty>>
+      <result> = shufflevector <vscale x n x <ty>> <v1>, <vscale x n x <ty>> v2, <vscale x m x i32> <mask>  ; yields <vscale x m x <ty>>
 
 Overview:
 """""""""
@@ -8249,6 +8269,10 @@ undef. If any element of the mask operand is undef, that element of the
 result is undef. If the shuffle mask selects an undef element from one
 of the input vectors, the resulting element is undef.
 
+For scalable vectors, the only valid mask values at present are
+``zeroinitializer`` and ``undef``, since we cannot write all indices as
+literals for a vector with a length unknown at compile time.
+
 Example:
 """"""""
 

include/llvm/ADT/DenseMapInfo.h

@@ -17,6 +17,7 @@
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/PointerLikeTypeTraits.h"
+#include "llvm/Support/ScalableSize.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
@@ -268,6 +269,21 @@ template <> struct DenseMapInfo<hash_code> {
   static bool isEqual(hash_code LHS, hash_code RHS) { return LHS == RHS; }
 };
 
+template <> struct DenseMapInfo<ElementCount> {
+  static inline ElementCount getEmptyKey() { return {~0U, true}; }
+  static inline ElementCount getTombstoneKey() { return {~0U - 1, false}; }
+  static unsigned getHashValue(const ElementCount& EltCnt) {
+    if (EltCnt.Scalable)
+      return (EltCnt.Min * 37U) - 1U;
+
+    return EltCnt.Min * 37U;
+  }
+
+  static bool isEqual(const ElementCount& LHS, const ElementCount& RHS) {
+    return LHS == RHS;
+  }
+};
+
 } // end namespace llvm
 
 #endif // LLVM_ADT_DENSEMAPINFO_H

include/llvm/IR/DerivedTypes.h

@@ -23,6 +23,7 @@
 #include "llvm/IR/Type.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/ScalableSize.h"
 #include <cassert>
 #include <cstdint>
 
@@ -387,6 +388,8 @@ public:
   SequentialType(const SequentialType &) = delete;
   SequentialType &operator=(const SequentialType &) = delete;
 
+  /// For scalable vectors, this will return the minimum number of elements
+  /// in the vector.
   uint64_t getNumElements() const { return NumElements; }
   Type *getElementType() const { return ContainedType; }
 
@@ -422,14 +425,37 @@ uint64_t Type::getArrayNumElements() const {
 
 /// Class to represent vector types.
 class VectorType : public SequentialType {
-  VectorType(Type *ElType, unsigned NumEl);
+  /// A fully specified VectorType is of the form <vscale x n x Ty>. 'n' is the
+  /// minimum number of elements of type Ty contained within the vector, and
+  /// 'scalable' indicates that the total element count is an integer multiple
+  /// of 'n', where the multiple is either guaranteed to be one, or is
+  /// statically unknown at compile time.
+  ///
+  /// If the multiple is known to be 1, then the extra term is discarded in
+  /// textual IR:
+  ///
+  /// <4 x i32>          - a vector containing 4 i32s
+  /// <vscale x 4 x i32> - a vector containing an unknown integer multiple
+  ///                      of 4 i32s
+
+  VectorType(Type *ElType, unsigned NumEl, bool Scalable = false);
+  VectorType(Type *ElType, ElementCount EC);
+
+  // If true, the total number of elements is an unknown multiple of the
+  // minimum 'NumElements' from SequentialType. Otherwise the total number
+  // of elements is exactly equal to 'NumElements'.
+  bool Scalable;
 
 public:
   VectorType(const VectorType &) = delete;
   VectorType &operator=(const VectorType &) = delete;
 
   /// This static method is the primary way to construct an VectorType.
-  static VectorType *get(Type *ElementType, unsigned NumElements);
+  static VectorType *get(Type *ElementType, ElementCount EC);
+  static VectorType *get(Type *ElementType, unsigned NumElements,
+                         bool Scalable = false) {
+    return VectorType::get(ElementType, {NumElements, Scalable});
+  }
 
   /// This static method gets a VectorType with the same number of elements as
   /// the input type, and the element type is an integer type of the same width
@@ -438,7 +464,7 @@ public:
     unsigned EltBits = VTy->getElementType()->getPrimitiveSizeInBits();
     assert(EltBits && "Element size must be of a non-zero size");
     Type *EltTy = IntegerType::get(VTy->getContext(), EltBits);
-    return VectorType::get(EltTy, VTy->getNumElements());
+    return VectorType::get(EltTy, VTy->getElementCount());
   }
 
   /// This static method is like getInteger except that the element types are
@@ -446,7 +472,7 @@ public:
   static VectorType *getExtendedElementVectorType(VectorType *VTy) {
     unsigned EltBits = VTy->getElementType()->getPrimitiveSizeInBits();
     Type *EltTy = IntegerType::get(VTy->getContext(), EltBits * 2);
-    return VectorType::get(EltTy, VTy->getNumElements());
+    return VectorType::get(EltTy, VTy->getElementCount());
   }
 
   /// This static method is like getInteger except that the element types are
@@ -456,29 +482,45 @@ public:
     assert((EltBits & 1) == 0 &&
            "Cannot truncate vector element with odd bit-width");
     Type *EltTy = IntegerType::get(VTy->getContext(), EltBits / 2);
-    return VectorType::get(EltTy, VTy->getNumElements());
+    return VectorType::get(EltTy, VTy->getElementCount());
   }
 
   /// This static method returns a VectorType with half as many elements as the
   /// input type and the same element type.
   static VectorType *getHalfElementsVectorType(VectorType *VTy) {
-    unsigned NumElts = VTy->getNumElements();
-    assert ((NumElts & 1) == 0 &&
+    auto EltCnt = VTy->getElementCount();
+    assert ((EltCnt.Min & 1) == 0 &&
             "Cannot halve vector with odd number of elements.");
-    return VectorType::get(VTy->getElementType(), NumElts/2);
+    return VectorType::get(VTy->getElementType(), EltCnt/2);
   }
 
   /// This static method returns a VectorType with twice as many elements as the
   /// input type and the same element type.
   static VectorType *getDoubleElementsVectorType(VectorType *VTy) {
-    unsigned NumElts = VTy->getNumElements();
-    return VectorType::get(VTy->getElementType(), NumElts*2);
+    auto EltCnt = VTy->getElementCount();
+    assert((VTy->getNumElements() * 2ull) <= UINT_MAX &&
+           "Too many elements in vector");
+    return VectorType::get(VTy->getElementType(), EltCnt*2);
   }
 
   /// Return true if the specified type is valid as a element type.
   static bool isValidElementType(Type *ElemTy);
 
-  /// Return the number of bits in the Vector type.
+  /// Return an ElementCount instance to represent the (possibly scalable)
+  /// number of elements in the vector.
+  ElementCount getElementCount() const {
+    uint64_t MinimumEltCnt = getNumElements();
+    assert(MinimumEltCnt <= UINT_MAX && "Too many elements in vector");
+    return { (unsigned)MinimumEltCnt, Scalable };
+  }
+
+  /// Returns whether or not this is a scalable vector (meaning the total
+  /// element count is a multiple of the minimum).
+  bool isScalable() const {
+    return Scalable;
+  }
+
+  /// Return the minimum number of bits in the Vector type.
   /// Returns zero when the vector is a vector of pointers.
   unsigned getBitWidth() const {
     return getNumElements() * getElementType()->getPrimitiveSizeInBits();
@@ -494,6 +536,10 @@ unsigned Type::getVectorNumElements() const {
   return cast<VectorType>(this)->getNumElements();
 }
 
+bool Type::getVectorIsScalable() const {
+  return cast<VectorType>(this)->isScalable();
+}
+
 /// Class to represent pointers.
 class PointerType : public Type {
   explicit PointerType(Type *ElType, unsigned AddrSpace);

include/llvm/IR/Type.h

@@ -366,6 +366,7 @@ public:
     return ContainedTys[0];
   }
 
+  inline bool getVectorIsScalable() const;
   inline unsigned getVectorNumElements() const;
   Type *getVectorElementType() const {
     assert(getTypeID() == VectorTyID);

include/llvm/Support/ScalableSize.h

@@ -0,0 +1,43 @@
+//===- ScalableSize.h - Scalable vector size info ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides a struct that can be used to query the size of IR types
+// which may be scalable vectors. It provides convenience operators so that
+// it can be used in much the same way as a single scalar value.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_SCALABLESIZE_H
+#define LLVM_SUPPORT_SCALABLESIZE_H
+
+namespace llvm {
+
+class ElementCount {
+public:
+  unsigned Min;  // Minimum number of vector elements.
+  bool Scalable; // If true, NumElements is a multiple of 'Min' determined
+                 // at runtime rather than compile time.
+
+  ElementCount(unsigned Min, bool Scalable)
+  : Min(Min), Scalable(Scalable) {}
+
+  ElementCount operator*(unsigned RHS) {
+    return { Min * RHS, Scalable };
+  }
+  ElementCount operator/(unsigned RHS) {
+    return { Min / RHS, Scalable };
+  }
+
+  bool operator==(const ElementCount& RHS) const {
+    return Min == RHS.Min && Scalable == RHS.Scalable;
+  }
+};
+
+} // end namespace llvm
+
+#endif // LLVM_SUPPORT_SCALABLESIZE_H

lib/AsmParser/LLLexer.cpp

@@ -707,6 +707,7 @@ lltok::Kind LLLexer::LexIdentifier() {
   KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
   KEYWORD(umin);
 
+  KEYWORD(vscale);
   KEYWORD(x);
   KEYWORD(blockaddress);
 

lib/AsmParser/LLParser.cpp

@@ -2721,7 +2721,18 @@ bool LLParser::ParseStructBody(SmallVectorImpl<Type*> &Body) {
 ///   Type
 ///     ::= '[' APSINTVAL 'x' Types ']'
 ///     ::= '<' APSINTVAL 'x' Types '>'
+///     ::= '<' 'vscale' 'x' APSINTVAL 'x' Types '>'
 bool LLParser::ParseArrayVectorType(Type *&Result, bool isVector) {
+  bool Scalable = false;
+
+  if (isVector && Lex.getKind() == lltok::kw_vscale) {
+    Lex.Lex(); // consume the 'vscale'
+    if (ParseToken(lltok::kw_x, "expected 'x' after vscale"))
+      return true;
+
+    Scalable = true;
+  }
+
   if (Lex.getKind() != lltok::APSInt || Lex.getAPSIntVal().isSigned() ||
       Lex.getAPSIntVal().getBitWidth() > 64)
     return TokError("expected number in address space");
@@ -2748,7 +2759,7 @@ bool LLParser::ParseArrayVectorType(Type *&Result, bool isVector) {
       return Error(SizeLoc, "size too large for vector");
     if (!VectorType::isValidElementType(EltTy))
       return Error(TypeLoc, "invalid vector element type");
-    Result = VectorType::get(EltTy, unsigned(Size));
+    Result = VectorType::get(EltTy, unsigned(Size), Scalable);
   } else {
     if (!ArrayType::isValidElementType(EltTy))
       return Error(TypeLoc, "invalid array element type");

lib/AsmParser/LLToken.h

@@ -37,6 +37,7 @@ enum Kind {
   bar,     // |
   colon,   // :
 
+  kw_vscale,
   kw_x,
   kw_true,
   kw_false,

lib/Bitcode/Reader/BitcodeReader.cpp

@@ -1758,7 +1758,8 @@ Error BitcodeReader::parseTypeTableBody() {
         return error("Invalid type");
       ResultTy = ArrayType::get(ResultTy, Record[0]);
       break;
-    case bitc::TYPE_CODE_VECTOR:    // VECTOR: [numelts, eltty]
+    case bitc::TYPE_CODE_VECTOR:    // VECTOR: [numelts, eltty] or
+                                    //         [numelts, eltty, scalable]
       if (Record.size() < 2)
         return error("Invalid record");
       if (Record[0] == 0)
@@ -1766,7 +1767,8 @@ Error BitcodeReader::parseTypeTableBody() {
       ResultTy = getTypeByID(Record[1]);
       if (!ResultTy || !StructType::isValidElementType(ResultTy))
         return error("Invalid type");
-      ResultTy = VectorType::get(ResultTy, Record[0]);
+      bool Scalable = Record.size() > 2 ? Record[2] : false;
+      ResultTy = VectorType::get(ResultTy, Record[0], Scalable);
       break;
     }
 

lib/Bitcode/Writer/BitcodeWriter.cpp

@@ -931,10 +931,13 @@ void ModuleBitcodeWriter::writeTypeTable() {
     }
     case Type::VectorTyID: {
       VectorType *VT = cast<VectorType>(T);
-      // VECTOR [numelts, eltty]
+      // VECTOR [numelts, eltty] or
+      //        [numelts, eltty, scalable]
       Code = bitc::TYPE_CODE_VECTOR;
       TypeVals.push_back(VT->getNumElements());
       TypeVals.push_back(VE.getTypeID(VT->getElementType()));
+      if (VT->isScalable())
+        TypeVals.push_back(VT->isScalable());
       break;
     }
     }

lib/IR/AsmWriter.cpp

@@ -620,7 +620,10 @@ void TypePrinting::print(Type *Ty, raw_ostream &OS) {
   }
   case Type::VectorTyID: {
     VectorType *PTy = cast<VectorType>(Ty);
-    OS << "<" << PTy->getNumElements() << " x ";
+    OS << "<";
+    if (PTy->isScalable())
+      OS << "vscale x ";
+    OS << PTy->getNumElements() << " x ";
     print(PTy->getElementType(), OS);
     OS << '>';
     return;

lib/IR/LLVMContextImpl.h

@@ -1334,7 +1334,7 @@ public:
   unsigned NamedStructTypesUniqueID = 0;
 
   DenseMap<std::pair<Type *, uint64_t>, ArrayType*> ArrayTypes;
-  DenseMap<std::pair<Type *, unsigned>, VectorType*> VectorTypes;
+  DenseMap<std::pair<Type *, ElementCount>, VectorType*> VectorTypes;
   DenseMap<Type*, PointerType*> PointerTypes;  // Pointers in AddrSpace = 0
   DenseMap<std::pair<Type*, unsigned>, PointerType*> ASPointerTypes;
 

lib/IR/Type.cpp

@@ -599,21 +599,20 @@ bool ArrayType::isValidElementType(Type *ElemTy) {
 //                          VectorType Implementation
 //===----------------------------------------------------------------------===//
 
-VectorType::VectorType(Type *ElType, unsigned NumEl)
-  : SequentialType(VectorTyID, ElType, NumEl) {}
+VectorType::VectorType(Type *ElType, ElementCount EC)
+  : SequentialType(VectorTyID, ElType, EC.Min), Scalable(EC.Scalable) {}
 
-VectorType *VectorType::get(Type *ElementType, unsigned NumElements) {
-  assert(NumElements > 0 && "#Elements of a VectorType must be greater than 0");
+VectorType *VectorType::get(Type *ElementType, ElementCount EC) {
+  assert(EC.Min > 0 && "#Elements of a VectorType must be greater than 0");
   assert(isValidElementType(ElementType) && "Element type of a VectorType must "
                                             "be an integer, floating point, or "
                                             "pointer type.");
 
   LLVMContextImpl *pImpl = ElementType->getContext().pImpl;
   VectorType *&Entry = ElementType->getContext().pImpl
-    ->VectorTypes[std::make_pair(ElementType, NumElements)];
-
+                                 ->VectorTypes[std::make_pair(ElementType, EC)];
   if (!Entry)
-    Entry = new (pImpl->Alloc) VectorType(ElementType, NumElements);
+    Entry = new (pImpl->Alloc) VectorType(ElementType, EC);
   return Entry;
 }
 

lib/IR/Verifier.cpp

@@ -43,6 +43,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "LLVMContextImpl.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
@@ -307,6 +308,7 @@ class Verifier : public InstVisitor<Verifier>, VerifierSupport {
   TBAAVerifier TBAAVerifyHelper;
 
   void checkAtomicMemAccessSize(Type *Ty, const Instruction *I);
+  static bool containsScalableVectorValue(const Type *Ty);
 
 public:
   explicit Verifier(raw_ostream *OS, bool ShouldTreatBrokenDebugInfoAsError,
@@ -318,6 +320,33 @@ public:
 
   bool hasBrokenDebugInfo() const { return BrokenDebugInfo; }
 
+  bool verifyTypes(const Module &M) {
+    LLVMContext &Ctx = M.getContext();
+    for (auto &Entry : Ctx.pImpl->ArrayTypes) {
+      ArrayType *ATy = Entry.second;
+      if (containsScalableVectorValue(ATy)) {
+        CheckFailed("Arrays cannot contain scalable vectors", ATy, &M);
+        Broken = true;
+      }
+    }
+
+    for (StructType* STy : Ctx.pImpl->AnonStructTypes)
+      if (containsScalableVectorValue(STy)) {
+        CheckFailed("Structs cannot contain scalable vectors", STy, &M);
+        Broken = true;
+      }
+
+    for (auto &Entry : Ctx.pImpl->NamedStructTypes) {
+      StructType *STy = Entry.second;
+      if (containsScalableVectorValue(STy)) {
+        CheckFailed("Structs cannot contain scalable vectors", STy, &M);
+        Broken = true;
+      }
+    }
+
+    return !Broken;
+  }
+
   bool verify(const Function &F) {
     assert(F.getParent() == &M &&
            "An instance of this class only works with a specific module!");
@@ -387,6 +416,8 @@ public:
 
     verifyCompileUnits();
 
+    verifyTypes(M);
+
     verifyDeoptimizeCallingConvs();
     DISubprogramAttachments.clear();
     return !Broken;
@@ -613,6 +644,35 @@ void Verifier::visitGlobalValue(const GlobalValue &GV) {
   });
 }
 
+// Check for a scalable vector type, making sure to look through arrays and
+// structs. Pointers to scalable vectors don't count, since we know what the
+// size of a pointer is.
+static bool containsScalableVectorValueRecursive(const Type *Ty,
+                                        SmallVectorImpl<const Type*> &Visited) {
+  if (is_contained(Visited, Ty))
+    return false;
+
+  Visited.push_back(Ty);
+
+  if (auto *VTy = dyn_cast<VectorType>(Ty))
+    return VTy->isScalable();
+
+  if (auto *ATy = dyn_cast<ArrayType>(Ty))
+    return containsScalableVectorValueRecursive(ATy->getElementType(), Visited);
+
+  if (auto *STy = dyn_cast<StructType>(Ty))
+    for (Type *EltTy : STy->elements())
+      if (containsScalableVectorValueRecursive(EltTy, Visited))
+        return true;
+
+  return false;
+}
+
+bool Verifier::containsScalableVectorValue(const Type *Ty) {
+  SmallVector<const Type*, 16> VisitedList = {};
+  return containsScalableVectorValueRecursive(Ty, VisitedList);
+}
+
 void Verifier::visitGlobalVariable(const GlobalVariable &GV) {
   if (GV.hasInitializer()) {
     Assert(GV.getInitializer()->getType() == GV.getValueType(),
@@ -691,6 +751,12 @@ void Verifier::visitGlobalVariable(const GlobalVariable &GV) {
                       "DIGlobalVariableExpression");
   }
 
+  // Scalable vectors cannot be global variables, since we don't know
+  // the runtime size. Need to look inside structs/arrays to find the
+  // underlying element type as well.
+  if (containsScalableVectorValue(GV.getValueType()))
+    CheckFailed("Globals cannot contain scalable vectors", &GV);
+
   if (!GV.hasInitializer()) {
     visitGlobalValue(GV);
     return;

test/Bitcode/compatibility.ll

@@ -917,6 +917,10 @@ define void @typesystem() {
   ; CHECK: %t7 = alloca x86_mmx
   %t8 = alloca %opaquety*
   ; CHECK: %t8 = alloca %opaquety*
+  %t9 = alloca <4 x i32>
+  ; CHECK: %t9 = alloca <4 x i32>
+  %t10 = alloca <vscale x 4 x i32>
+  ; CHECK: %t10 = alloca <vscale x 4 x i32>
 
   ret void
 }

test/Verifier/scalable-aggregates.ll

@@ -0,0 +1,31 @@
+; RUN: not opt -S -verify < %s 2>&1 | FileCheck %s
+
+;; Arrays and Structs cannot contain scalable vectors, since we don't
+;; know the size at compile time and the container types need to have
+;; a known size.
+
+; CHECK-DAG: Arrays cannot contain scalable vectors
+; CHECK-DAG:  [2 x { i32, <vscale x 1 x i32> }]; ModuleID = '<stdin>'
+; CHECK-DAG: Arrays cannot contain scalable vectors
+; CHECK-DAG:  [4 x <vscale x 256 x i1>]; ModuleID = '<stdin>'
+; CHECK-DAG: Arrays cannot contain scalable vectors
+; CHECK-DAG:  [2 x <vscale x 4 x i32>]; ModuleID = '<stdin>'
+; CHECK-DAG: Structs cannot contain scalable vectors
+; CHECK-DAG:  { i64, [4 x <vscale x 256 x i1>] }; ModuleID = '<stdin>'
+; CHECK-DAG: Structs cannot contain scalable vectors
+; CHECK-DAG:  { i32, <vscale x 1 x i32> }; ModuleID = '<stdin>'
+; CHECK-DAG: Structs cannot contain scalable vectors
+; CHECK-DAG: { <vscale x 16 x i8>, <vscale x 2 x double> }; ModuleID = '<stdin>'
+; CHECK-DAG: Structs cannot contain scalable vectors
+; CHECK-DAG:  %sty = type { i64, <vscale x 32 x i16> }; ModuleID = '<stdin>'
+
+%sty = type { i64, <vscale x 32 x i16> }
+
+define void @scalable_aggregates() {
+  %array = alloca [2 x <vscale x 4 x i32>]
+  %struct = alloca { <vscale x 16 x i8>, <vscale x 2 x double> }
+  %named_struct = alloca %sty
+  %s_in_a = alloca [2 x { i32, <vscale x 1 x i32> } ]
+  %a_in_s = alloca { i64, [4 x <vscale x 256 x i1> ] }
+  ret void
+}

test/Verifier/scalable-global-vars.ll

@@ -0,0 +1,24 @@
+; RUN: not opt -S -verify < %s 2>&1 | FileCheck %s
+
+;; Global variables cannot be scalable vectors, since we don't
+;; know the size at compile time.
+
+; CHECK: Globals cannot contain scalable vectors
+; CHECK-NEXT: <vscale x 4 x i32>* @ScalableVecGlobal
+@ScalableVecGlobal = global <vscale x 4 x i32> zeroinitializer
+
+; CHECK: Globals cannot contain scalable vectors
+; CHECK-NEXT: [64 x <vscale x 2 x double>]* @ScalableVecGlobalArray
+@ScalableVecGlobalArray = global [64 x <vscale x 2 x double>] zeroinitializer
+
+; CHECK: Globals cannot contain scalable vectors
+; CHECK-NEXT: { <vscale x 16 x i64>, <vscale x 16 x i1> }* @ScalableVecGlobalStruct
+@ScalableVecGlobalStruct = global { <vscale x 16 x i64>, <vscale x 16 x i1> } zeroinitializer
+
+; CHECK: Globals cannot contain scalable vectors
+; CHECK-NEXT: { [4 x i32], [2 x { <vscale x 4 x i64>, <vscale x 32 x i8> }] }* @ScalableVecMixed
+@ScalableVecMixed = global { [4 x i32], [2 x { <vscale x 4 x i64>,  <vscale x 32 x i8> }]} zeroinitializer
+
+;; Global _pointers_ to scalable vectors are fine
+; CHECK-NOT: Globals cannot contain scalable vectors
+@ScalableVecPtr = global <vscale x 8 x i16>* zeroinitializer

unittests/IR/CMakeLists.txt

@@ -37,6 +37,7 @@ add_llvm_unittest(IRTests
   ValueHandleTest.cpp
   ValueMapTest.cpp
   ValueTest.cpp
+  VectorTypesTest.cpp
   VerifierTest.cpp
   WaymarkTest.cpp
   )

unittests/IR/VectorTypesTest.cpp

@@ -0,0 +1,164 @@
+//===--- llvm/unittest/IR/VectorTypesTest.cpp - vector types unit tests ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/DerivedTypes.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/Support/ScalableSize.h"
+#include "gtest/gtest.h"
+using namespace llvm;
+
+namespace {
+TEST(VectorTypesTest, FixedLength) {
+  LLVMContext Ctx;
+
+  Type *Int16Ty = Type::getInt16Ty(Ctx);
+  Type *Int32Ty = Type::getInt32Ty(Ctx);
+  Type *Int64Ty = Type::getInt64Ty(Ctx);
+  Type *Float64Ty = Type::getDoubleTy(Ctx);
+
+  VectorType *V8Int32Ty = VectorType::get(Int32Ty, 8);
+  ASSERT_FALSE(V8Int32Ty->isScalable());
+  EXPECT_EQ(V8Int32Ty->getNumElements(), 8U);
+  EXPECT_EQ(V8Int32Ty->getElementType()->getScalarSizeInBits(), 32U);
+
+  VectorType *V8Int16Ty = VectorType::get(Int16Ty, {8, false});
+  ASSERT_FALSE(V8Int16Ty->isScalable());
+  EXPECT_EQ(V8Int16Ty->getNumElements(), 8U);
+  EXPECT_EQ(V8Int16Ty->getElementType()->getScalarSizeInBits(), 16U);
+
+  ElementCount EltCnt(4, false);
+  VectorType *V4Int64Ty = VectorType::get(Int64Ty, EltCnt);
+  ASSERT_FALSE(V4Int64Ty->isScalable());
+  EXPECT_EQ(V4Int64Ty->getNumElements(), 4U);
+  EXPECT_EQ(V4Int64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *V2Int64Ty = VectorType::get(Int64Ty, EltCnt/2);
+  ASSERT_FALSE(V2Int64Ty->isScalable());
+  EXPECT_EQ(V2Int64Ty->getNumElements(), 2U);
+  EXPECT_EQ(V2Int64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *V8Int64Ty = VectorType::get(Int64Ty, EltCnt*2);
+  ASSERT_FALSE(V8Int64Ty->isScalable());
+  EXPECT_EQ(V8Int64Ty->getNumElements(), 8U);
+  EXPECT_EQ(V8Int64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *V4Float64Ty = VectorType::get(Float64Ty, EltCnt);
+  ASSERT_FALSE(V4Float64Ty->isScalable());
+  EXPECT_EQ(V4Float64Ty->getNumElements(), 4U);
+  EXPECT_EQ(V4Float64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ExtTy = VectorType::getExtendedElementVectorType(V8Int16Ty);
+  EXPECT_EQ(ExtTy, V8Int32Ty);
+  ASSERT_FALSE(ExtTy->isScalable());
+  EXPECT_EQ(ExtTy->getNumElements(), 8U);
+  EXPECT_EQ(ExtTy->getElementType()->getScalarSizeInBits(), 32U);
+
+  VectorType *TruncTy = VectorType::getTruncatedElementVectorType(V8Int32Ty);
+  EXPECT_EQ(TruncTy, V8Int16Ty);
+  ASSERT_FALSE(TruncTy->isScalable());
+  EXPECT_EQ(TruncTy->getNumElements(), 8U);
+  EXPECT_EQ(TruncTy->getElementType()->getScalarSizeInBits(), 16U);
+
+  VectorType *HalvedTy = VectorType::getHalfElementsVectorType(V4Int64Ty);
+  EXPECT_EQ(HalvedTy, V2Int64Ty);
+  ASSERT_FALSE(HalvedTy->isScalable());
+  EXPECT_EQ(HalvedTy->getNumElements(), 2U);
+  EXPECT_EQ(HalvedTy->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *DoubledTy = VectorType::getDoubleElementsVectorType(V4Int64Ty);
+  EXPECT_EQ(DoubledTy, V8Int64Ty);
+  ASSERT_FALSE(DoubledTy->isScalable());
+  EXPECT_EQ(DoubledTy->getNumElements(), 8U);
+  EXPECT_EQ(DoubledTy->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ConvTy = VectorType::getInteger(V4Float64Ty);
+  EXPECT_EQ(ConvTy, V4Int64Ty);
+  ASSERT_FALSE(ConvTy->isScalable());
+  EXPECT_EQ(ConvTy->getNumElements(), 4U);
+  EXPECT_EQ(ConvTy->getElementType()->getScalarSizeInBits(), 64U);
+
+  EltCnt = V8Int64Ty->getElementCount();
+  EXPECT_EQ(EltCnt.Min, 8U);
+  ASSERT_FALSE(EltCnt.Scalable);
+}
+
+TEST(VectorTypesTest, Scalable) {
+  LLVMContext Ctx;
+
+  Type *Int16Ty = Type::getInt16Ty(Ctx);
+  Type *Int32Ty = Type::getInt32Ty(Ctx);
+  Type *Int64Ty = Type::getInt64Ty(Ctx);
+  Type *Float64Ty = Type::getDoubleTy(Ctx);
+
+  VectorType *ScV8Int32Ty = VectorType::get(Int32Ty, 8, true);
+  ASSERT_TRUE(ScV8Int32Ty->isScalable());
+  EXPECT_EQ(ScV8Int32Ty->getNumElements(), 8U);
+  EXPECT_EQ(ScV8Int32Ty->getElementType()->getScalarSizeInBits(), 32U);
+
+  VectorType *ScV8Int16Ty = VectorType::get(Int16Ty, {8, true});
+  ASSERT_TRUE(ScV8Int16Ty->isScalable());
+  EXPECT_EQ(ScV8Int16Ty->getNumElements(), 8U);
+  EXPECT_EQ(ScV8Int16Ty->getElementType()->getScalarSizeInBits(), 16U);
+
+  ElementCount EltCnt(4, true);
+  VectorType *ScV4Int64Ty = VectorType::get(Int64Ty, EltCnt);
+  ASSERT_TRUE(ScV4Int64Ty->isScalable());
+  EXPECT_EQ(ScV4Int64Ty->getNumElements(), 4U);
+  EXPECT_EQ(ScV4Int64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ScV2Int64Ty = VectorType::get(Int64Ty, EltCnt/2);
+  ASSERT_TRUE(ScV2Int64Ty->isScalable());
+  EXPECT_EQ(ScV2Int64Ty->getNumElements(), 2U);
+  EXPECT_EQ(ScV2Int64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ScV8Int64Ty = VectorType::get(Int64Ty, EltCnt*2);
+  ASSERT_TRUE(ScV8Int64Ty->isScalable());
+  EXPECT_EQ(ScV8Int64Ty->getNumElements(), 8U);
+  EXPECT_EQ(ScV8Int64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ScV4Float64Ty = VectorType::get(Float64Ty, EltCnt);
+  ASSERT_TRUE(ScV4Float64Ty->isScalable());
+  EXPECT_EQ(ScV4Float64Ty->getNumElements(), 4U);
+  EXPECT_EQ(ScV4Float64Ty->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ExtTy = VectorType::getExtendedElementVectorType(ScV8Int16Ty);
+  EXPECT_EQ(ExtTy, ScV8Int32Ty);
+  ASSERT_TRUE(ExtTy->isScalable());
+  EXPECT_EQ(ExtTy->getNumElements(), 8U);
+  EXPECT_EQ(ExtTy->getElementType()->getScalarSizeInBits(), 32U);
+
+  VectorType *TruncTy = VectorType::getTruncatedElementVectorType(ScV8Int32Ty);
+  EXPECT_EQ(TruncTy, ScV8Int16Ty);
+  ASSERT_TRUE(TruncTy->isScalable());
+  EXPECT_EQ(TruncTy->getNumElements(), 8U);
+  EXPECT_EQ(TruncTy->getElementType()->getScalarSizeInBits(), 16U);
+
+  VectorType *HalvedTy = VectorType::getHalfElementsVectorType(ScV4Int64Ty);
+  EXPECT_EQ(HalvedTy, ScV2Int64Ty);
+  ASSERT_TRUE(HalvedTy->isScalable());
+  EXPECT_EQ(HalvedTy->getNumElements(), 2U);
+  EXPECT_EQ(HalvedTy->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *DoubledTy = VectorType::getDoubleElementsVectorType(ScV4Int64Ty);
+  EXPECT_EQ(DoubledTy, ScV8Int64Ty);
+  ASSERT_TRUE(DoubledTy->isScalable());
+  EXPECT_EQ(DoubledTy->getNumElements(), 8U);
+  EXPECT_EQ(DoubledTy->getElementType()->getScalarSizeInBits(), 64U);
+
+  VectorType *ConvTy = VectorType::getInteger(ScV4Float64Ty);
+  EXPECT_EQ(ConvTy, ScV4Int64Ty);
+  ASSERT_TRUE(ConvTy->isScalable());
+  EXPECT_EQ(ConvTy->getNumElements(), 4U);
+  EXPECT_EQ(ConvTy->getElementType()->getScalarSizeInBits(), 64U);
+
+  EltCnt = ScV8Int64Ty->getElementCount();
+  EXPECT_EQ(EltCnt.Min, 8U);
+  ASSERT_TRUE(EltCnt.Scalable);
+}
+
+} // end anonymous namespace