Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X........X...............X] duration: 5.448359301s FAIL (2025-08-11 03:28:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 03:28:45.940590727 (58h2m40.371670848s) [upload=60506]: Starting upload Aug 11 03:28:45.940661994 (58h2m40.371742115s) [upload=60506]: Done starting upload Aug 11 03:28:45.940747237 (58h2m40.371827368s) [upload=60506]: Uploading segments... Aug 11 03:28:45.940842238 (58h2m40.371922369s) [upload=60506]: Got next segment. Inline: false Aug 11 03:28:45.940867386 (58h2m40.371947517s) [upload=60506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:28:45.940871274 (58h2m40.371951405s) [upload=60506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:28:46.386421262 (58h2m40.817501393s) [seg_pos={0 0}, upload=60506]: Begin upload segment... Aug 11 03:28:46.386462240 (58h2m40.817542361s) [seg_pos={0 0}, upload=60506]: Done begin upload segment. Aug 11 03:28:46.386483331 (58h2m40.817563462s) [noise=true , address=104.36.82.43:4859 , piece_id=FE7T3SNBFEJT3P763MTAGATBYO7QNWV67J3I3Q24O7MCYPNJP2NQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.386566189 (58h2m40.81764632s) [noise=true , address=104.36.82.36:4646 , piece_id=MMJYJULKREEQ2NKZQSNUASMH7G4LECAT3NJHP5Y4SCZWLCNCJONQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.386832249 (58h2m40.81791238s) [noise=true , address=104.36.82.35:4613 , piece_id=CAH3YTS5O27DKAASH37TNSXAIK72GGR7TLKP4YXDKBBI3XCAEG7A, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.387095041 (58h2m40.818175172s) [noise=true , address=79.127.201.215:4341 , piece_id=MG4ARXQCOSRUU6ZW5Q37BHETNS5IHFX752GS46QYI4SQ3AEGFIZA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.387239477 (58h2m40.818319608s) [noise=true , address=156.146.43.225:4442 , piece_id=BSEL3MS3BPL6EXPXKVZDIRJEVEY3F5LVM2Q7NJFQBDMYZMDPEXBA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.387357203 (58h2m40.818437334s) [noise=true , address=104.36.82.62:4417 , piece_id=OENCOR2CZNPHFM5774IHU6GMYA4HPUZF7TL3WGJICAWOR37VLWEA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.387625747 (58h2m40.818705868s) [noise=true , address=104.36.82.46:4511 , piece_id=ZMURBCUDR74DDHNRFCU4X5I7QUCVFKM2TV4TQGLVKSOWVDUUFZPA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.387763531 (58h2m40.818843652s) [noise=true , address=104.36.82.8:4978 , piece_id=H2ONVG5KEIGO36Y5SEU2HOB55WBJJC36LC63UC2C7LLB467UDRQA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.387847481 (58h2m40.818927612s) [noise=true , address=104.36.82.2:4750 , piece_id=54LPIX4UOXPPPZKUTTKOZIGNFKUCUO7M2J2RZ5NYUHFAW3USSBYA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388072391 (58h2m40.819152512s) [noise=true , address=104.36.82.44:4872 , piece_id=GGP72S2TFAPA6X446AEGLLD6OA27EQ4ZGLRRJAFGNV2G3EHG5OIQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388108961 (58h2m40.819189092s) [noise=true , address=104.36.82.61:4373 , piece_id=N2NY3W5NPQPLBCM7EZYXXNCYTIHQI2M4VOOKZPAEZWLDOL5T64GA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388384759 (58h2m40.81946488s) [noise=true , address=64.49.13.169:4716 , piece_id=3EVL5CEWHMKZQTGGRDY65EQDF4VCOTBG436YFARS7SGVDTCS5NYA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388518224 (58h2m40.819598355s) [noise=true , address=104.36.82.41:4770 , piece_id=7YQYROKTIDZ4XCY4EV5UWW5VQVGYSIZBO23RH7TOQSUB3PGXN2WQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388634607 (58h2m40.819714728s) [noise=true , address=104.36.82.18:4379 , piece_id=RYUUWYK2U4GG3NE3MRD6Q2FDSN3M4RF2JQGDNDMD7YFHFKIX26UQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388845460 (58h2m40.819925591s) [noise=true , address=79.127.205.246:4257 , piece_id=3OEXD75LHOX3GKNCFJSTRFALZBZMY25Q7BK5QVXBY2BSX6TGORBQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.388859998 (58h2m40.819940129s) [noise=true , address=104.36.82.23:4468 , piece_id=HEYMQ3V3RWCC5YJAJSTVBNENS5PQYZP23PQVQOIXA722MXDJCSUA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.389197424 (58h2m40.820277555s) [noise=true , address=104.36.82.60:4351 , piece_id=MRGZJP3B2OVM6XIFFXCOTRCVPSOS4UQPFXQVPZ3AEO5AKONN6DMQ, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:46.389508348 (58h2m40.82058848s) [noise=true , address=104.36.82.19:4404 , piece_id=SPU4KJ2YD4NWUOQNYZ6M42GFS6AIOOJHB6PYQLHTV5ZXCXC2QUKA, seg_pos={0 0}, upload=60506]: Uploading piece... Aug 11 03:28:47.572715210 (58h2m42.003795341s) [upload=60506]: Next returned nil segment Aug 11 03:28:47.572729709 (58h2m42.00380984s) [upload=60506]: Waiting for error group managing segments... Aug 11 03:28:47.872595432 (58h2m42.303675563s) [noise=true , address=79.127.205.246:4257 , piece_id=3OEXD75LHOX3GKNCFJSTRFALZBZMY25Q7BK5QVXBY2BSX6TGORBQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:48.068847247 (58h2m42.499927368s) [noise=true , address=64.49.13.169:4716 , piece_id=3EVL5CEWHMKZQTGGRDY65EQDF4VCOTBG436YFARS7SGVDTCS5NYA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:48.328455536 (58h2m42.759535667s) [noise=true , address=156.146.43.225:4442 , piece_id=BSEL3MS3BPL6EXPXKVZDIRJEVEY3F5LVM2Q7NJFQBDMYZMDPEXBA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:48.337222981 (58h2m42.768303112s) [noise=true , address=79.127.201.215:4341 , piece_id=MG4ARXQCOSRUU6ZW5Q37BHETNS5IHFX752GS46QYI4SQ3AEGFIZA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:48.674099429 (58h2m43.10517956s) [noise=true , address=104.36.82.41:4770 , piece_id=7YQYROKTIDZ4XCY4EV5UWW5VQVGYSIZBO23RH7TOQSUB3PGXN2WQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:48.757106369 (58h2m43.1881865s) [noise=true , address=104.36.82.61:4373 , piece_id=N2NY3W5NPQPLBCM7EZYXXNCYTIHQI2M4VOOKZPAEZWLDOL5T64GA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:48.769217631 (58h2m43.200297762s) [noise=true , address=104.36.82.43:4859 , piece_id=FE7T3SNBFEJT3P763MTAGATBYO7QNWV67J3I3Q24O7MCYPNJP2NQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.214957131 (58h2m43.646037262s) [noise=true , address=104.36.82.18:4379 , piece_id=RYUUWYK2U4GG3NE3MRD6Q2FDSN3M4RF2JQGDNDMD7YFHFKIX26UQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.356739478 (58h2m43.787819609s) [noise=true , address=104.36.82.62:4417 , piece_id=OENCOR2CZNPHFM5774IHU6GMYA4HPUZF7TL3WGJICAWOR37VLWEA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.437004232 (58h2m43.868084363s) [noise=true , address=104.36.82.2:4750 , piece_id=54LPIX4UOXPPPZKUTTKOZIGNFKUCUO7M2J2RZ5NYUHFAW3USSBYA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.504046316 (58h2m43.935126437s) [noise=true , address=104.36.82.44:4872 , piece_id=GGP72S2TFAPA6X446AEGLLD6OA27EQ4ZGLRRJAFGNV2G3EHG5OIQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.610960097 (58h2m44.042040228s) [noise=true , address=104.36.82.19:4404 , piece_id=SPU4KJ2YD4NWUOQNYZ6M42GFS6AIOOJHB6PYQLHTV5ZXCXC2QUKA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.630928513 (58h2m44.062008644s) [noise=true , address=104.36.82.8:4978 , piece_id=H2ONVG5KEIGO36Y5SEU2HOB55WBJJC36LC63UC2C7LLB467UDRQA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.698423233 (58h2m44.129503354s) [noise=true , address=104.36.82.23:4468 , piece_id=HEYMQ3V3RWCC5YJAJSTVBNENS5PQYZP23PQVQOIXA722MXDJCSUA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: <nil> Aug 11 03:28:49.698447790 (58h2m44.129527911s) [seg_pos={0 0}, upload=60506]: Segment reached optimal threshold of 14 pieces. Aug 11 03:28:49.698552932 (58h2m44.129633063s) [noise=true , address=104.36.82.36:4646 , piece_id=MMJYJULKREEQ2NKZQSNUASMH7G4LECAT3NJHP5Y4SCZWLCNCJONQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zA8TqySnmGc9bVQnNNMSZEPbv9Gf1Wa1BwYrKjc2ZGgtyEmrh): protocol: expected piece hash; context canceled; EOF Aug 11 03:28:49.698610211 (58h2m44.129690342s) [noise=true , address=104.36.82.46:4511 , piece_id=ZMURBCUDR74DDHNRFCU4X5I7QUCVFKM2TV4TQGLVKSOWVDUUFZPA, seg_pos={0 0}, upload=60506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Aug 11 03:28:49.698623507 (58h2m44.129703638s) [noise=true , address=104.36.82.35:4613 , piece_id=CAH3YTS5O27DKAASH37TNSXAIK72GGR7TLKP4YXDKBBI3XCAEG7A, seg_pos={0 0}, upload=60506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xjm7XPPwM9nXBP3SS36HeLNxBVJdEpcQyD9BAp79xbGCze4vL): protocol: expected piece hash; context canceled; EOF Aug 11 03:28:49.698688511 (58h2m44.129768642s) [noise=true , address=104.36.82.60:4351 , piece_id=MRGZJP3B2OVM6XIFFXCOTRCVPSOS4UQPFXQVPZ3AEO5AKONN6DMQ, seg_pos={0 0}, upload=60506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; write tcp 188.166.225.170:35254->104.36.82.60:4351: use of closed network connection; write tcp 188.166.225.170:35254->104.36.82.60:4351: use of closed network connection; EOF Aug 11 03:28:49.698701597 (58h2m44.129781728s) [seg_pos={0 0}, upload=60506]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 03:28:49.698756101 (58h2m44.129836222s) [upload=60506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 03:28:49.698759928 (58h2m44.129840059s) [upload=60506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:28:50.084800474 (58h2m44.515880605s) [upload=60506]: Done uploading segments... Aug 11 03:28:50.084819189 (58h2m44.51589932s) [upload=60506]: Upload finished. err: <nil> Duration: 5.448386473s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration: 6.307008265s FAIL (2025-08-11 03:28:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.307108907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [X.X...X...X.................X....X.............X............] duration: 1.86820812s FAIL (2025-08-11 03:28:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.86823922s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X........X...............] duration: 7.519740602s FAIL (2025-08-11 03:28:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.519768074s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X....X..............] duration: 8.724491924s FAIL (2025-08-11 03:27:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.724513495s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.X.X...X...X.................X....X.............X...........] duration: 1.859704249s FAIL (2025-08-11 03:27:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.859739675s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X........X..............] duration: 8.045778959s FAIL (2025-08-11 03:27:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 8.045802163s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X........X.............] duration: 7.808326039s FAIL (2025-08-11 03:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.808355706s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration: 5.93799947s FAIL (2025-08-11 03:26:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.938016574s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [..X.X...X...X.................X....X.............X..........] duration: 1.838823147s FAIL (2025-08-11 03:26:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.838859536s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X........X............] duration: 8.132253353s FAIL (2025-08-11 03:26:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 8.132285333s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 6.537156827s FAIL (2025-08-11 03:25:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.537180624s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [...X.X...X...X.................X....X.............X.........] duration: 1.876405012s FAIL (2025-08-11 03:25:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.876430651s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 5.948133176s FAIL (2025-08-11 03:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.948157262s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration: 7.950115232s FAIL (2025-08-11 03:25:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.950168275s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 5.913387536s FAIL (2025-08-11 03:24:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.913407966s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [X...X.X...X...X.................X....X.............X........] duration: 1.811008649s FAIL (2025-08-11 03:24:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.81105677s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 6.196482133s FAIL (2025-08-11 03:24:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.196511179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X........X..........] duration: 8.085771696s FAIL (2025-08-11 03:24:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 8.085803084s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.X...X.X...X...X.................X....X.............X.......] duration: 1.829268292s FAIL (2025-08-11 03:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.829291667s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 6.058765765s FAIL (2025-08-11 03:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.058787907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration: 7.42421446s FAIL (2025-08-11 03:23:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.424242574s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration: 6.513427538s FAIL (2025-08-11 03:23:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.513451794s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [..X...X.X...X...X.................X....X.............X......] duration: 1.834008192s FAIL (2025-08-11 03:22:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.83403761s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X........X........] duration: 7.45798412s FAIL (2025-08-11 03:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.458009418s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [...X...X.X...X...X.................X....X.............X.....] duration: 1.8521641s FAIL (2025-08-11 03:22:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.852206191s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X....X......] duration: 6.424209455s FAIL (2025-08-11 03:22:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.424230726s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration: 7.717149478s FAIL (2025-08-11 03:22:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.71718695s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X........X......] duration: 9.965293127s FAIL (2025-08-11 03:21:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 9.965315038s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [....X...X.X...X...X.................X....X.............X....] duration: 1.982722029s FAIL (2025-08-11 03:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.982748579s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration: 11.42848485s FAIL (2025-08-11 03:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 11.428508624s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration: 6.602983443s FAIL (2025-08-11 03:21:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.603004716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.....X...X.X...X...X.................X....X.............X...] duration: 1.909991253s FAIL (2025-08-11 03:21:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.910043694s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X....X....] duration: 6.056374009s FAIL (2025-08-11 03:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.056396532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X........X....] duration: 7.733875174s FAIL (2025-08-11 03:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 7.733901765s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X....X...] duration: 5.785327666s FAIL (2025-08-11 03:20:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.785345732s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 8.033257195s FAIL (2025-08-11 03:20:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 8.033284328s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [......X...X.X...X...X.................X....X.............X..] duration: 1.82993061s FAIL (2025-08-11 03:19:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.829963403s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X....X..] duration: 6.803130408s FAIL (2025-08-11 03:19:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.803154526s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.......X...X.X...X...X.................X....X.............X.] duration: 1.820342806s FAIL (2025-08-11 03:18:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.820368947s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 8.725944266s FAIL (2025-08-11 03:18:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 8.725966118s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 30.075418714s FAIL (2025-08-11 03:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 30.075449834s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 9.146716809s FAIL (2025-08-11 03:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.146737929s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.000540317s FAIL (2025-08-11 03:18:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000558672s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [........X...X.X...X...X.................X....X.............X] duration: 1.603518812s FAIL (2025-08-11 03:18:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 509b491301fe4a2414703b6765da796dd548f9d2c6b24ad3529ce3351b142193 != before 91135a8730ae486bfd4199b43ecc728ba6e645e71181b3c44236f1d881e4575b ======LOGS====== Aug 11 03:18:11.216659207 (57h52m5.647739338s) [upload=60319]: Starting upload Aug 11 03:18:11.216716727 (57h52m5.647796858s) [upload=60319]: Done starting upload Aug 11 03:18:11.216782834 (57h52m5.647862965s) [upload=60319]: Uploading segments... Aug 11 03:18:11.216821648 (57h52m5.647901769s) [upload=60319]: Got next segment. Inline: true Aug 11 03:18:11.216825275 (57h52m5.647905396s) [upload=60319]: Waiting for error group managing segments... Aug 11 03:18:11.216857146 (57h52m5.647937277s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:18:11.216861033 (57h52m5.647941164s) [upload=60319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:18:11.216863869 (57h52m5.647944001s) [upload=60319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:18:11.492059343 (57h52m5.923139485s) [upload=60319]: Done uploading segments... Aug 11 03:18:11.492080735 (57h52m5.923160856s) [upload=60319]: Upload finished. err: <nil> Duration:* 1.603543439s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.496098574s FAIL (2025-08-11 03:17:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.496125034s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X.X...X...X.................X....X.............] duration: 1.948272033s FAIL (2025-08-11 03:17:31 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 11 03:08:03.426831847 (57h41m57.857911978s) [upload=60145]: Starting upload Aug 11 03:08:03.426885058 (57h41m57.857965189s) [upload=60145]: Done starting upload Aug 11 03:08:03.426943391 (57h41m57.858023522s) [upload=60145]: Uploading segments... Aug 11 03:08:03.426983517 (57h41m57.858063638s) [upload=60145]: Got next segment. Inline: true Aug 11 03:08:03.426987204 (57h41m57.858067325s) [upload=60145]: Waiting for error group managing segments... Aug 11 03:08:03.427018704 (57h41m57.858098825s) [upload=60145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:08:03.427022702 (57h41m57.858102823s) [upload=60145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:08:03.427025418 (57h41m57.858105549s) [upload=60145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:08:04.171547230 (57h41m58.602627361s) [upload=60145]: Done uploading segments... Aug 11 03:08:04.171566748 (57h41m58.602646879s) [upload=60145]: Upload finished. err: <nil> Duration:* 1.948298734s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 851.947791ms FAIL (2025-08-11 03:17:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 03:17:30.597443340 (57h51m25.028523471s) [upload=60306]: Starting upload Aug 11 03:17:30.597507203 (57h51m25.028587344s) [upload=60306]: Done starting upload Aug 11 03:17:30.597604448 (57h51m25.028684579s) [upload=60306]: Uploading segments... Aug 11 03:17:30.597740348 (57h51m25.028820479s) [upload=60306]: Got next segment. Inline: false Aug 11 03:17:30.597786196 (57h51m25.028866337s) [upload=60306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:17:30.597791897 (57h51m25.028872028s) [upload=60306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:17:30.809449083 (57h51m25.240529214s) [upload=60306]: Next segment err: metaclient: Too Many Requests Aug 11 03:17:30.809470564 (57h51m25.240550685s) [upload=60306]: Done uploading segments... Aug 11 03:17:30.809483088 (57h51m25.240563209s) [upload=60306]: Upload finished. err: metaclient: Too Many Requests Duration: 851.971798ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.378656576s FAIL (2025-08-11 03:17:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 03:10:19.695831728 (57h44m14.126911859s) [upload=60185]: Starting upload Aug 11 03:10:19.695893396 (57h44m14.126973527s) [upload=60185]: Done starting upload Aug 11 03:10:19.696003306 (57h44m14.127083427s) [upload=60185]: Uploading segments... Aug 11 03:10:19.696123055 (57h44m14.127203186s) [upload=60185]: Got next segment. Inline: false Aug 11 03:10:19.696168853 (57h44m14.127248984s) [upload=60185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:10:19.696173952 (57h44m14.127254083s) [upload=60185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 03:10:20.141758356 (57h44m14.572838487s) [seg_pos={0 0}, upload=60185]: Begin upload segment... Aug 11 03:10:20.141842587 (57h44m14.572922718s) [seg_pos={0 0}, upload=60185]: Done begin upload segment. Aug 11 03:10:20.141866302 (57h44m14.572946433s) [noise=true , address=104.36.82.52:4091 , piece_id=M5AJ46RHG6JZNKAPEKODYT4MHA4SHQJSBOUI6VRDYA2QI6SWI6DA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.141976904 (57h44m14.573057035s) [noise=true , address=104.36.82.35:4598 , piece_id=CIDU6ZPHD5OQ3K3SKDGYBF24AIKL3R74I7NR2S4DC643BQHB7Y5A, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.142251820 (57h44m14.573331951s) [noise=true , address=104.36.82.44:4882 , piece_id=L3O23BBCIYOGLP6ZQPKYUW34ATBKGH7KFIC6TQW4Z7FE2AUT4MLA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.142470739 (57h44m14.57355087s) [noise=true , address=104.36.82.37:4671 , piece_id=UGFZ2XY5ROFAGLLO3ITU7XZDXWWOJY4CXYIKCUBOZM5ZOSLH4SOA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.142543158 (57h44m14.573623289s) [noise=true , address=104.36.82.59:4329 , piece_id=MS2D3G2WB6GW6MMO56LIX45NVPAPQFYEGDGGBTBYT352CD33KSZA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.142766555 (57h44m14.573846686s) [noise=true , address=79.127.201.215:4348 , piece_id=SML3QA63TMKLTG74YICPVEVM52232JRXH3V42FSVVBVHAOXMHKSQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.142828764 (57h44m14.573908895s) [noise=true , address=104.36.82.61:4384 , piece_id=FOTSODG2SSNGA4Z6ZDYMRSB72NGGFYBSJK3M3HOUJZRSLKHJPACA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143055437 (57h44m14.574135578s) [noise=true , address=104.36.82.37:4677 , piece_id=OZ26VWEUC7J6AUPZWCKL6HVS3R4CXYEAOKEHIEVLTZPTDH6SFIHQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143095174 (57h44m14.574175305s) [noise=true , address=104.36.82.22:4509 , piece_id=WT4KZHIEBVJPGADUD4UVGPCDRR2RC3YTFKQ7GOBD3GKI57TT5C7Q, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143340533 (57h44m14.574420664s) [noise=true , address=64.49.13.26:4945 , piece_id=WGVRIXR5B6DZLF3N5LMZKUI434CFLSLFMBYMC72XWPNPPVROQKPQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143350903 (57h44m14.574431034s) [noise=true , address=104.36.82.18:4368 , piece_id=E5QI3AGHINSRSHLE4I7HDEU6FHVIFJVDAIG4MOZ5CRBRVRIXZCUQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143575673 (57h44m14.574655804s) [noise=true , address=104.36.82.47:4549 , piece_id=TUH5LJ5LK4DJU2MYMLZFXU7XEAWXV2BGNWODMXX7EO4CQA6Y4U2A, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143609186 (57h44m14.574689318s) [noise=true , address=104.36.82.13:4216 , piece_id=NFRAEX73C46CPO25NSQC2KYHQ2DQVC5IFED3HMPS3EVSP4RBT4ZA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143858493 (57h44m14.574938624s) [noise=true , address=104.36.82.38:4705 , piece_id=XXVKP7V5FPHAQBXUWUECEQNC3LK4GAWWQH77R2DXNEKSTKLWFXHQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.143909531 (57h44m14.574989662s) [noise=true , address=104.36.82.12:4182 , piece_id=V44M4DSO6Y32BLY4BTDWCCP6ZSAMTDURKQRKWBIKASMCMZBCNXWQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.144108342 (57h44m14.575188473s) [noise=true , address=104.36.82.23:4468 , piece_id=CGD547YA44S4DIGTZ6U6BTKAFXG246HFBBNCEMZT352FRV5FOGMQ, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.144168227 (57h44m14.575248358s) [noise=true , address=79.127.219.38:4033 , piece_id=FLZKKUOK5MFMSO3MMBCRMYRGTA3NQAAUHTBWXIDCBLPNNJ3ECXCA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:20.144371375 (57h44m14.575451506s) [noise=true , address=104.36.82.2:4753 , piece_id=UIDURIUZDKLAOVIIH3GFZLSDB5OVV5NYWAOIFF5NMCYT2CSBW7CA, seg_pos={0 0}, upload=60185]: Uploading piece... Aug 11 03:10:21.444260910 (57h44m15.875341041s) [upload=60185]: Next returned nil segment Aug 11 03:10:21.444338608 (57h44m15.875418739s) [upload=60185]: Waiting for error group managing segments... Aug 11 03:10:21.727033111 (57h44m16.158113242s) [noise=true , address=64.49.13.26:4945 , piece_id=WGVRIXR5B6DZLF3N5LMZKUI434CFLSLFMBYMC72XWPNPPVROQKPQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:22.079132658 (57h44m16.510212789s) [noise=true , address=79.127.201.215:4348 , piece_id=SML3QA63TMKLTG74YICPVEVM52232JRXH3V42FSVVBVHAOXMHKSQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:22.117350672 (57h44m16.548430803s) [noise=true , address=79.127.219.38:4033 , piece_id=FLZKKUOK5MFMSO3MMBCRMYRGTA3NQAAUHTBWXIDCBLPNNJ3ECXCA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:22.959169063 (57h44m17.390249194s) [noise=true , address=104.36.82.35:4598 , piece_id=CIDU6ZPHD5OQ3K3SKDGYBF24AIKL3R74I7NR2S4DC643BQHB7Y5A, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:23.078803401 (57h44m17.509883532s) [noise=true , address=104.36.82.61:4384 , piece_id=FOTSODG2SSNGA4Z6ZDYMRSB72NGGFYBSJK3M3HOUJZRSLKHJPACA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:23.280703499 (57h44m17.71178363s) [noise=true , address=104.36.82.47:4549 , piece_id=TUH5LJ5LK4DJU2MYMLZFXU7XEAWXV2BGNWODMXX7EO4CQA6Y4U2A, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:23.307502814 (57h44m17.738582935s) [noise=true , address=104.36.82.23:4468 , piece_id=CGD547YA44S4DIGTZ6U6BTKAFXG246HFBBNCEMZT352FRV5FOGMQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:24.237523806 (57h44m18.668603927s) [noise=true , address=104.36.82.38:4705 , piece_id=XXVKP7V5FPHAQBXUWUECEQNC3LK4GAWWQH77R2DXNEKSTKLWFXHQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:24.277731726 (57h44m18.708811847s) [noise=true , address=104.36.82.13:4216 , piece_id=NFRAEX73C46CPO25NSQC2KYHQ2DQVC5IFED3HMPS3EVSP4RBT4ZA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:25.075167210 (57h44m19.506247341s) [noise=true , address=104.36.82.37:4671 , piece_id=UGFZ2XY5ROFAGLLO3ITU7XZDXWWOJY4CXYIKCUBOZM5ZOSLH4SOA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:26.989360971 (57h44m21.420441102s) [noise=true , address=104.36.82.37:4677 , piece_id=OZ26VWEUC7J6AUPZWCKL6HVS3R4CXYEAOKEHIEVLTZPTDH6SFIHQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:28.993998460 (57h44m23.425078591s) [noise=true , address=104.36.82.52:4091 , piece_id=M5AJ46RHG6JZNKAPEKODYT4MHA4SHQJSBOUI6VRDYA2QI6SWI6DA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:29.267665081 (57h44m23.698745222s) [noise=true , address=104.36.82.12:4182 , piece_id=V44M4DSO6Y32BLY4BTDWCCP6ZSAMTDURKQRKWBIKASMCMZBCNXWQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:30.080531428 (57h44m24.511611549s) [noise=true , address=104.36.82.44:4882 , piece_id=L3O23BBCIYOGLP6ZQPKYUW34ATBKGH7KFIC6TQW4Z7FE2AUT4MLA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: <nil> Aug 11 03:10:30.080556265 (57h44m24.511636396s) [seg_pos={0 0}, upload=60185]: Segment reached optimal threshold of 14 pieces. Aug 11 03:10:30.080704869 (57h44m24.511785s) [noise=true , address=104.36.82.59:4329 , piece_id=MS2D3G2WB6GW6MMO56LIX45NVPAPQFYEGDGGBTBYT352CD33KSZA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dGY6BG4FULjuRwnEKd9wV4cZPVJsgo32oFHKZn5BgjxVaojbk): protocol: expected piece hash; context canceled; EOF Aug 11 03:10:30.080789982 (57h44m24.511870103s) [noise=true , address=104.36.82.22:4509 , piece_id=WT4KZHIEBVJPGADUD4UVGPCDRR2RC3YTFKQ7GOBD3GKI57TT5C7Q, seg_pos={0 0}, upload=60185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:50436->104.36.82.22:4509: use of closed network connection; write tcp 188.166.225.170:50436->104.36.82.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50436->104.36.82.22:4509: use of closed network connection Aug 11 03:10:30.080873071 (57h44m24.511953222s) [noise=true , address=104.36.82.18:4368 , piece_id=E5QI3AGHINSRSHLE4I7HDEU6FHVIFJVDAIG4MOZ5CRBRVRIXZCUQ, seg_pos={0 0}, upload=60185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ByhoW6Effjs8acfsvdF1xzDwBVqggsBK53oRZ8bYhEYYPChKz): protocol: expected piece hash; context canceled; EOF Aug 11 03:10:30.080932976 (57h44m24.512013108s) [noise=true , address=104.36.82.2:4753 , piece_id=UIDURIUZDKLAOVIIH3GFZLSDB5OVV5NYWAOIFF5NMCYT2CSBW7CA, seg_pos={0 0}, upload=60185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu): protocol: expected piece hash; context canceled; EOF Aug 11 03:10:30.080954176 (57h44m24.512034307s) [seg_pos={0 0}, upload=60185]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 03:10:30.081015084 (57h44m24.512095215s) [upload=60185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 03:10:30.081018630 (57h44m24.512098761s) [upload=60185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:10:30.396534371 (57h44m24.827614502s) [upload=60185]: Done uploading segments... Aug 11 03:10:30.396552516 (57h44m24.827632647s) [upload=60185]: Upload finished. err: <nil> Duration: 8.378694919s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [X.........X...X.X...X...X.................X....X............] duration: 1.815504309s FAIL (2025-08-11 03:17:08 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 11 03:08:03.426831847 (57h41m57.857911978s) [upload=60145]: Starting upload Aug 11 03:08:03.426885058 (57h41m57.857965189s) [upload=60145]: Done starting upload Aug 11 03:08:03.426943391 (57h41m57.858023522s) [upload=60145]: Uploading segments... Aug 11 03:08:03.426983517 (57h41m57.858063638s) [upload=60145]: Got next segment. Inline: true Aug 11 03:08:03.426987204 (57h41m57.858067325s) [upload=60145]: Waiting for error group managing segments... Aug 11 03:08:03.427018704 (57h41m57.858098825s) [upload=60145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 03:08:03.427022702 (57h41m57.858102823s) [upload=60145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 03:08:03.427025418 (57h41m57.858105549s) [upload=60145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 03:08:04.171547230 (57h41m58.602627361s) [upload=60145]: Done uploading segments... Aug 11 03:08:04.171566748 (57h41m58.602646879s) [upload=60145]: Upload finished. err: <nil> Duration:* 1.815535167s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)