CTO News Hubb
Advertisement
  • Home
  • CTO News
  • IT
  • Technology
  • Tech Topics
    • AI
    • QC
    • Robotics
    • Blockchain
  • Contact
No Result
View All Result
  • Home
  • CTO News
  • IT
  • Technology
  • Tech Topics
    • AI
    • QC
    • Robotics
    • Blockchain
  • Contact
No Result
View All Result
CTO News Hubb
No Result
View All Result
Home CTO News

OpenOrca

June 29, 2023
in CTO News


Today I am announcing OpenOrca, an open-source dataset and series of instruct-tuned language models.

As I read Orca: Progressive Learning from Complex Explanation Traces of GPT-4 by Mukherjee et. al. of Microsoft, I had to consider the implications for Open Source AI.

This was pretty awesome stuff. But, I realized that while Microsoft would probably release their LLaMA-13b based model (as of the time of this writing they still haven’t) I concluded that they might not release the dataset.

Therefore, I resolved to replicate their efforts, download the data myself, and train the model myself, so that OpenOrca can be released on other sizes of LLaMA as well as other foundational models such as Falcon, OpenLLaMA, RedPajama, MPT, RWKV.

This was a nontrivial undertaking. With the help of an all-star team of open-source AI/ML engineers, we have completed the OpenOrca dataset.

Our dataset consists of:

We followed the submix and system prompt distribution outlined in the Orca paper. With a few exceptions. We included all 75k of CoT in the FLAN-1m dataset rather than sampling that. Also, we found that many items were duplicated so we removed duplicates, resulting in 3.5m instructs in the ChatGPT dataset.

We are presently performing full weights fine-tuning of OpenOrca on the foundation of LLaMA-13b, so that our performance can be compared with Microsoft’s model when it releases.

We expect to release OpenOrca-LLaMA-13b in mid-July 2023. At that time we will publish our evaluation findings and the dataset.

We are currently seeking GPU compute sponsors for training OpenOrca on the following platforms:

From the Orca paper and our experiments, we roughly estimate the compute costs as follows:

Model Size Compute Estimate
7b 1k GPU-Hours
13b 2k GPU-Hours
30/33b 4k-6k GPU-Hours
40b 8k-10k GPU-Hours
65b 10k-15k GPU-Hours

We will share our appreciation for sponsorship in this space, as well as the model cards.

Our current sponsors:

Please reach out to me if you are interested in providing compute sponsorship for any specific targets of OpenOrca.

I would like to thank the motley crew of Open Source AI/ML engineers who have worked beside me in this endeavor. Including:

  • Wing “Caseus” Lian and NanoBit of OpenAccess AI Collective

  • AutoMeta, Entropi, AtlasUnified, and neverendingtoast of Alignment Lab AI

  • Rohan

  • Teknium

  • Pankaj Mathur

  • Tom “TheBloke” Jobbins for quantizing and amplifying

  • All the other people in the Open Source AI community who have taught me and helped me along the way.



Source link

Previous Post

Rust language gets new governance

Next Post

Announcement – The Most Awaited ChatGPT Fundamentals Course Launched

Next Post

Announcement - The Most Awaited ChatGPT Fundamentals Course Launched

Joanne Pransky: Rest in Peace (1959-2023)

Trending News

Quality of new vehicles in US declining on more tech use, study shows

June 23, 2023

OPNsense® a true open source security platform and more

June 27, 2023

Our journey at F5 with Apache Arrow (part 2): Adaptive Schemas and Sorting to Optimize Arrow Usage

July 5, 2023

© CTO News Hubb All rights reserved.

Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Privacy Policy and Terms & Conditions.

Navigate Site

  • Home
  • CTO News
  • IT
  • Technology
  • AI
  • QC
  • Robotics
  • Blockchain
  • Contact

Newsletter Sign Up

No Result
View All Result
  • Home
  • CTO News
  • IT
  • Technology
  • Tech Topics
    • AI
    • QC
    • Robotics
    • Blockchain
  • Contact

© 2021 JNews – Premium WordPress news & magazine theme by Jegtheme.

SUBSCRIBE TO OUR WEEKLY NEWSLETTERS