Upload tokenizer

README.md

@@ -6,29 +6,66 @@ tags:
 - NLP
 pipeline_tag: summarization
 widget:
-- text: 
-    ' Moderator: Welcome, everyone, to this exciting panel discussion. Today, we have Elon Musk and Sam Altman, two of the most influential figures in the tech industry. We’re here to discuss the future of artificial intelligence and its impact on society. Elon, Sam, thank you for joining us.
-    Elon Musk: Happy to be here.
-    Sam Altman: Looking forward to the discussion.
-    Moderator: Let’s dive right in. Elon, you’ve been very vocal about your concerns regarding AI. Could you elaborate on why you believe AI poses such a significant risk to humanity?
-    Elon Musk: Certainly. AI has the potential to become more intelligent than humans, which could be extremely dangerous if it goes unchecked. The existential threat is real. If we don’t implement strict regulations and oversight, we risk creating something that could outsmart us and act against our interests. It’s a ticking time bomb.
-    Sam Altman: I respect Elon’s concerns, but I think he’s overestimating the threat. The focus should be on leveraging AI to solve some of humanity’s biggest problems. With proper ethical frameworks and robust safety measures, we can ensure AI benefits everyone. The fear-mongering is unproductive and could hinder technological progress.
-    Elon Musk: It’s not fear-mongering, Sam. It’s being cautious. We need to ensure that we have control mechanisms in place. Without these, we’re playing with fire. You can’t possibly believe that AI will always remain benevolent or under our control.
-    Sam Altman: Control mechanisms are essential, I agree, but what you’re suggesting sounds like stifling innovation out of fear. We need a balanced approach. Overregulation could slow down advancements that could otherwise save lives and improve quality of life globally. We must foster innovation while ensuring safety, not let fear dictate our actions.
-    Elon Musk: Balancing innovation and safety is easier said than done. When you’re dealing with something as unpredictable and powerful as AI, the risks far outweigh the potential benefits if we don’t tread carefully. History has shown us the dangers of underestimating new technologies.
-    Sam Altman: And history has also shown us the incredible benefits of technological advancement. If we had been overly cautious, we might not have the medical, communication, or energy technologies we have today. It’s about finding that middle ground where innovation thrives safely. We can’t just halt progress because of hypothetical risks.
-    Elon Musk: It’s not hypothetical, Sam. Look at how quickly AI capabilities are advancing. We’re already seeing issues with bias, decision-making, and unintended consequences. Imagine this on a larger scale. We can’t afford to be complacent.
-    Sam Altman: Bias and unintended consequences are exactly why we need to invest in research and development to address these issues head-on. By building AI responsibly and learning from each iteration, we can mitigate these risks. Shutting down or heavily regulating AI development out of fear isn’t the solution.
-    Moderator: Both of you make compelling points. Let’s fast forward a bit. Say, ten years from now, we have stringent regulations in place, as Elon suggests, or a more flexible framework, as Sam proposes. What does the world look like?
-    Elon Musk: With stringent regulations, we would have a more controlled and safer AI development environment. This would prevent any catastrophic events and ensure that AI works for us, not against us. We’d be able to avoid many potential disasters that an unchecked AI might cause.
-    Sam Altman: On the other hand, with a more flexible framework, we’d see rapid advancements in AI applications across various sectors, from healthcare to education, bringing significant improvements to quality of life and solving problems that seem insurmountable today. The world would be a much better place with these innovations.
-    Moderator: And what if both of you are wrong?
-    Elon Musk: Wrong?
-    Sam Altman: How so?
-    Moderator: Suppose the future shows that neither stringent regulations nor a flexible framework were the key factors. Instead, what if the major breakthroughs and safety measures came from unexpected areas like quantum computing advancements or new forms of human-computer symbiosis, rendering this entire debate moot?
-    Elon Musk: Well, that’s a possibility. If breakthroughs in quantum computing or other technologies overshadow our current AI concerns, it could change the entire landscape. It’s difficult to predict all variables.
-    Sam Altman: Agreed. Technology often takes unexpected turns. If future advancements make our current debate irrelevant, it just goes to show how unpredictable and fast-moving the tech world is. The key takeaway would be the importance of adaptability and continuous learning.
-    Moderator: Fascinating. It appears that the only certainty in the tech world is uncertainty itself. Thank you both for this engaging discussion.'
+- text: ' Moderator: Welcome, everyone, to this exciting panel discussion. Today,
+    we have Elon Musk and Sam Altman, two of the most influential figures in the tech
+    industry. We’re here to discuss the future of artificial intelligence and its
+    impact on society. Elon, Sam, thank you for joining us. Elon Musk: Happy to be
+    here. Sam Altman: Looking forward to the discussion. Moderator: Let’s dive right
+    in. Elon, you’ve been very vocal about your concerns regarding AI. Could you elaborate
+    on why you believe AI poses such a significant risk to humanity? Elon Musk: Certainly.
+    AI has the potential to become more intelligent than humans, which could be extremely
+    dangerous if it goes unchecked. The existential threat is real. If we don’t implement
+    strict regulations and oversight, we risk creating something that could outsmart
+    us and act against our interests. It’s a ticking time bomb. Sam Altman: I respect
+    Elon’s concerns, but I think he’s overestimating the threat. The focus should
+    be on leveraging AI to solve some of humanity’s biggest problems. With proper
+    ethical frameworks and robust safety measures, we can ensure AI benefits everyone.
+    The fear-mongering is unproductive and could hinder technological progress. Elon
+    Musk: It’s not fear-mongering, Sam. It’s being cautious. We need to ensure that
+    we have control mechanisms in place. Without these, we’re playing with fire. You
+    can’t possibly believe that AI will always remain benevolent or under our control.
+    Sam Altman: Control mechanisms are essential, I agree, but what you’re suggesting
+    sounds like stifling innovation out of fear. We need a balanced approach. Overregulation
+    could slow down advancements that could otherwise save lives and improve quality
+    of life globally. We must foster innovation while ensuring safety, not let fear
+    dictate our actions. Elon Musk: Balancing innovation and safety is easier said
+    than done. When you’re dealing with something as unpredictable and powerful as
+    AI, the risks far outweigh the potential benefits if we don’t tread carefully.
+    History has shown us the dangers of underestimating new technologies. Sam Altman:
+    And history has also shown us the incredible benefits of technological advancement.
+    If we had been overly cautious, we might not have the medical, communication,
+    or energy technologies we have today. It’s about finding that middle ground where
+    innovation thrives safely. We can’t just halt progress because of hypothetical
+    risks. Elon Musk: It’s not hypothetical, Sam. Look at how quickly AI capabilities
+    are advancing. We’re already seeing issues with bias, decision-making, and unintended
+    consequences. Imagine this on a larger scale. We can’t afford to be complacent.
+    Sam Altman: Bias and unintended consequences are exactly why we need to invest
+    in research and development to address these issues head-on. By building AI responsibly
+    and learning from each iteration, we can mitigate these risks. Shutting down or
+    heavily regulating AI development out of fear isn’t the solution. Moderator: Both
+    of you make compelling points. Let’s fast forward a bit. Say, ten years from now,
+    we have stringent regulations in place, as Elon suggests, or a more flexible framework,
+    as Sam proposes. What does the world look like? Elon Musk: With stringent regulations,
+    we would have a more controlled and safer AI development environment. This would
+    prevent any catastrophic events and ensure that AI works for us, not against us.
+    We’d be able to avoid many potential disasters that an unchecked AI might cause.
+    Sam Altman: On the other hand, with a more flexible framework, we’d see rapid
+    advancements in AI applications across various sectors, from healthcare to education,
+    bringing significant improvements to quality of life and solving problems that
+    seem insurmountable today. The world would be a much better place with these innovations.
+    Moderator: And what if both of you are wrong? Elon Musk: Wrong? Sam Altman: How
+    so? Moderator: Suppose the future shows that neither stringent regulations nor
+    a flexible framework were the key factors. Instead, what if the major breakthroughs
+    and safety measures came from unexpected areas like quantum computing advancements
+    or new forms of human-computer symbiosis, rendering this entire debate moot? Elon
+    Musk: Well, that’s a possibility. If breakthroughs in quantum computing or other
+    technologies overshadow our current AI concerns, it could change the entire landscape.
+    It’s difficult to predict all variables. Sam Altman: Agreed. Technology often
+    takes unexpected turns. If future advancements make our current debate irrelevant,
+    it just goes to show how unpredictable and fast-moving the tech world is. The
+    key takeaway would be the importance of adaptability and continuous learning.
+    Moderator: Fascinating. It appears that the only certainty in the tech world is
+    uncertainty itself. Thank you both for this engaging discussion.'
   example_title: Sample 1
 ---
 # Arc of the Conversation Model

tokenizer.json

@@ -1,11 +1,6 @@
 {
   "version": "1.0",
-  "truncation": {
-    "direction": "Right",
-    "max_length": 1024,
-    "strategy": "LongestFirst",
-    "stride": 0
-  },
+  "truncation": null,
   "padding": null,
   "added_tokens": [
     {

tokenizer_config.json

@@ -930,7 +930,7 @@
   "clean_up_tokenization_spaces": true,
   "eos_token": "</s>",
   "extra_ids": 100,
-  "max_length": 1024,
+  "max_length": 2048,
   "model_max_length": 512,
   "pad_token": "<pad>",
   "stride": 0,