Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................................X....] duration: 6.673560537s FAIL (2025-08-14 23:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 14 23:08:35.105900497 (149h42m29.536980628s) [upload=155706]: Starting upload Aug 14 23:08:35.105956424 (149h42m29.537036555s) [upload=155706]: Done starting upload Aug 14 23:08:35.106058068 (149h42m29.537138199s) [upload=155706]: Uploading segments... Aug 14 23:08:35.106266597 (149h42m29.537346728s) [upload=155706]: Got next segment. Inline: false Aug 14 23:08:35.106292647 (149h42m29.537372778s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:08:35.106296905 (149h42m29.537377036s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 23:08:35.664735283 (149h42m30.095815414s) [seg_pos={0 0}, upload=155706]: Begin upload segment... Aug 14 23:08:35.664826968 (149h42m30.095907099s) [seg_pos={0 0}, upload=155706]: Done begin upload segment. Aug 14 23:08:35.664850333 (149h42m30.095930454s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.664953391 (149h42m30.096033522s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665339700 (149h42m30.096419831s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665585180 (149h42m30.096665311s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665729355 (149h42m30.096809486s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666053666 (149h42m30.097133797s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666335725 (149h42m30.097415896s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666464402 (149h42m30.097544533s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666808389 (149h42m30.09788852s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666990067 (149h42m30.098070208s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667066173 (149h42m30.098146304s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667286054 (149h42m30.098366185s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667410852 (149h42m30.098490983s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667548656 (149h42m30.098628787s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667718280 (149h42m30.098798401s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668161589 (149h42m30.09924171s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668464248 (149h42m30.099544379s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668807865 (149h42m30.099887996s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:36.779607273 (149h42m31.210687404s) [upload=155706]: Next returned nil segment Aug 14 23:08:36.779883893 (149h42m31.210964014s) [upload=155706]: Waiting for error group managing segments... Aug 14 23:08:37.080252078 (149h42m31.511332209s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.163526031 (149h42m31.594606162s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.201950862 (149h42m31.633030993s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.655662129 (149h42m32.08674226s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.686468463 (149h42m32.117548594s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.714936814 (149h42m32.146016945s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.751955404 (149h42m32.183035525s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.766303347 (149h42m32.197383478s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.819588991 (149h42m32.250669123s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.865209390 (149h42m32.296289521s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.257586512 (149h42m32.688666663s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.438386389 (149h42m32.86946652s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.502515471 (149h42m32.933595593s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826762315 (149h42m33.257842446s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826789567 (149h42m33.257869698s) [seg_pos={0 0}, upload=155706]: Segment reached optimal threshold of 14 pieces. Aug 14 23:08:38.826996513 (149h42m33.258076644s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: rpc: dial tcp 104.36.82.46:4502: operation was canceled; conn closed: op Aug 14 23:08:38.827180856 (149h42m33.258260977s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms): protocol: read tcp 188.166.225.170:35194->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827248545 (149h42m33.258328666s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection Aug 14 23:08:38.827256730 (149h42m33.258336862s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: read tcp 188.166.225.170:38054->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827279524 (149h42m33.258359655s) [seg_pos={0 0}, upload=155706]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 23:08:38.827354087 (149h42m33.258434218s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 23:08:38.827361080 (149h42m33.258441211s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:08:39.769810673 (149h42m34.200890804s) [upload=155706]: Done uploading segments... Aug 14 23:08:39.769827726 (149h42m34.200907857s) [upload=155706]: Upload finished. err: <nil> Duration: 6.673653375s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....X.......X...................................X.......X...] duration: 3.593330502s FAIL (2025-08-14 23:11:47 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 14 23:09:30.496311157 (149h43m24.927391288s) [upload=155721]: Starting upload Aug 14 23:09:30.496358919 (149h43m24.92743905s) [upload=155721]: Done starting upload Aug 14 23:09:30.496409976 (149h43m24.927490107s) [upload=155721]: Uploading segments... Aug 14 23:09:30.496446436 (149h43m24.927526567s) [upload=155721]: Got next segment. Inline: true Aug 14 23:09:30.496450264 (149h43m24.927530395s) [upload=155721]: Waiting for error group managing segments... Aug 14 23:09:30.496483657 (149h43m24.927563788s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:09:30.496487354 (149h43m24.927567485s) [upload=155721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:09:30.496489648 (149h43m24.927569779s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:09:33.939233724 (149h43m28.370313845s) [upload=155721]: Done uploading segments... Aug 14 23:09:33.939252460 (149h43m28.370332591s) [upload=155721]: Upload finished. err: <nil> Duration:* 3.593361941s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 752.872821ms FAIL (2025-08-14 23:11:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 752.906987ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 5.71289433s FAIL (2025-08-14 23:11:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.712931171s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.312939586s FAIL (2025-08-14 23:11:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.313008447s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................................X...] duration: 7.339509318s FAIL (2025-08-14 23:11:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 14 23:08:35.105900497 (149h42m29.536980628s) [upload=155706]: Starting upload Aug 14 23:08:35.105956424 (149h42m29.537036555s) [upload=155706]: Done starting upload Aug 14 23:08:35.106058068 (149h42m29.537138199s) [upload=155706]: Uploading segments... Aug 14 23:08:35.106266597 (149h42m29.537346728s) [upload=155706]: Got next segment. Inline: false Aug 14 23:08:35.106292647 (149h42m29.537372778s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:08:35.106296905 (149h42m29.537377036s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 23:08:35.664735283 (149h42m30.095815414s) [seg_pos={0 0}, upload=155706]: Begin upload segment... Aug 14 23:08:35.664826968 (149h42m30.095907099s) [seg_pos={0 0}, upload=155706]: Done begin upload segment. Aug 14 23:08:35.664850333 (149h42m30.095930454s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.664953391 (149h42m30.096033522s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665339700 (149h42m30.096419831s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665585180 (149h42m30.096665311s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665729355 (149h42m30.096809486s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666053666 (149h42m30.097133797s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666335725 (149h42m30.097415896s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666464402 (149h42m30.097544533s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666808389 (149h42m30.09788852s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666990067 (149h42m30.098070208s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667066173 (149h42m30.098146304s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667286054 (149h42m30.098366185s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667410852 (149h42m30.098490983s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667548656 (149h42m30.098628787s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667718280 (149h42m30.098798401s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668161589 (149h42m30.09924171s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668464248 (149h42m30.099544379s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668807865 (149h42m30.099887996s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:36.779607273 (149h42m31.210687404s) [upload=155706]: Next returned nil segment Aug 14 23:08:36.779883893 (149h42m31.210964014s) [upload=155706]: Waiting for error group managing segments... Aug 14 23:08:37.080252078 (149h42m31.511332209s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.163526031 (149h42m31.594606162s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.201950862 (149h42m31.633030993s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.655662129 (149h42m32.08674226s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.686468463 (149h42m32.117548594s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.714936814 (149h42m32.146016945s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.751955404 (149h42m32.183035525s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.766303347 (149h42m32.197383478s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.819588991 (149h42m32.250669123s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.865209390 (149h42m32.296289521s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.257586512 (149h42m32.688666663s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.438386389 (149h42m32.86946652s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.502515471 (149h42m32.933595593s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826762315 (149h42m33.257842446s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826789567 (149h42m33.257869698s) [seg_pos={0 0}, upload=155706]: Segment reached optimal threshold of 14 pieces. Aug 14 23:08:38.826996513 (149h42m33.258076644s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: rpc: dial tcp 104.36.82.46:4502: operation was canceled; conn closed: op Aug 14 23:08:38.827180856 (149h42m33.258260977s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms): protocol: read tcp 188.166.225.170:35194->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827248545 (149h42m33.258328666s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection Aug 14 23:08:38.827256730 (149h42m33.258336862s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: read tcp 188.166.225.170:38054->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827279524 (149h42m33.258359655s) [seg_pos={0 0}, upload=155706]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 23:08:38.827354087 (149h42m33.258434218s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 23:08:38.827361080 (149h42m33.258441211s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:08:39.769810673 (149h42m34.200890804s) [upload=155706]: Done uploading segments... Aug 14 23:08:39.769827726 (149h42m34.200907857s) [upload=155706]: Upload finished. err: <nil> Duration: 7.339534117s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X.......X...................................X.......X..] duration: 2.9328803s FAIL (2025-08-14 23:11:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 14 23:09:30.496311157 (149h43m24.927391288s) [upload=155721]: Starting upload Aug 14 23:09:30.496358919 (149h43m24.92743905s) [upload=155721]: Done starting upload Aug 14 23:09:30.496409976 (149h43m24.927490107s) [upload=155721]: Uploading segments... Aug 14 23:09:30.496446436 (149h43m24.927526567s) [upload=155721]: Got next segment. Inline: true Aug 14 23:09:30.496450264 (149h43m24.927530395s) [upload=155721]: Waiting for error group managing segments... Aug 14 23:09:30.496483657 (149h43m24.927563788s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:09:30.496487354 (149h43m24.927567485s) [upload=155721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:09:30.496489648 (149h43m24.927569779s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:09:33.939233724 (149h43m28.370313845s) [upload=155721]: Done uploading segments... Aug 14 23:09:33.939252460 (149h43m28.370332591s) [upload=155721]: Upload finished. err: <nil> Duration:* 2.932900669s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 780.754912ms FAIL (2025-08-14 23:10:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 780.781103ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 5.810257143s FAIL (2025-08-14 23:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.810280518s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration: 5.931581077s FAIL (2025-08-14 23:10:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 14 23:08:35.105900497 (149h42m29.536980628s) [upload=155706]: Starting upload Aug 14 23:08:35.105956424 (149h42m29.537036555s) [upload=155706]: Done starting upload Aug 14 23:08:35.106058068 (149h42m29.537138199s) [upload=155706]: Uploading segments... Aug 14 23:08:35.106266597 (149h42m29.537346728s) [upload=155706]: Got next segment. Inline: false Aug 14 23:08:35.106292647 (149h42m29.537372778s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:08:35.106296905 (149h42m29.537377036s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 23:08:35.664735283 (149h42m30.095815414s) [seg_pos={0 0}, upload=155706]: Begin upload segment... Aug 14 23:08:35.664826968 (149h42m30.095907099s) [seg_pos={0 0}, upload=155706]: Done begin upload segment. Aug 14 23:08:35.664850333 (149h42m30.095930454s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.664953391 (149h42m30.096033522s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665339700 (149h42m30.096419831s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665585180 (149h42m30.096665311s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665729355 (149h42m30.096809486s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666053666 (149h42m30.097133797s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666335725 (149h42m30.097415896s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666464402 (149h42m30.097544533s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666808389 (149h42m30.09788852s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666990067 (149h42m30.098070208s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667066173 (149h42m30.098146304s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667286054 (149h42m30.098366185s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667410852 (149h42m30.098490983s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667548656 (149h42m30.098628787s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667718280 (149h42m30.098798401s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668161589 (149h42m30.09924171s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668464248 (149h42m30.099544379s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668807865 (149h42m30.099887996s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:36.779607273 (149h42m31.210687404s) [upload=155706]: Next returned nil segment Aug 14 23:08:36.779883893 (149h42m31.210964014s) [upload=155706]: Waiting for error group managing segments... Aug 14 23:08:37.080252078 (149h42m31.511332209s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.163526031 (149h42m31.594606162s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.201950862 (149h42m31.633030993s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.655662129 (149h42m32.08674226s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.686468463 (149h42m32.117548594s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.714936814 (149h42m32.146016945s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.751955404 (149h42m32.183035525s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.766303347 (149h42m32.197383478s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.819588991 (149h42m32.250669123s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.865209390 (149h42m32.296289521s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.257586512 (149h42m32.688666663s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.438386389 (149h42m32.86946652s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.502515471 (149h42m32.933595593s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826762315 (149h42m33.257842446s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826789567 (149h42m33.257869698s) [seg_pos={0 0}, upload=155706]: Segment reached optimal threshold of 14 pieces. Aug 14 23:08:38.826996513 (149h42m33.258076644s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: rpc: dial tcp 104.36.82.46:4502: operation was canceled; conn closed: op Aug 14 23:08:38.827180856 (149h42m33.258260977s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms): protocol: read tcp 188.166.225.170:35194->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827248545 (149h42m33.258328666s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection Aug 14 23:08:38.827256730 (149h42m33.258336862s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: read tcp 188.166.225.170:38054->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827279524 (149h42m33.258359655s) [seg_pos={0 0}, upload=155706]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 23:08:38.827354087 (149h42m33.258434218s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 23:08:38.827361080 (149h42m33.258441211s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:08:39.769810673 (149h42m34.200890804s) [upload=155706]: Done uploading segments... Aug 14 23:08:39.769827726 (149h42m34.200907857s) [upload=155706]: Upload finished. err: <nil> Duration: 5.931606788s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.443720297s FAIL (2025-08-14 23:10:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.443744906s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 715.536003ms FAIL (2025-08-14 23:10:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 715.562755ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 752.335696ms FAIL (2025-08-14 23:10:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 752.375853ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X.......X...................................X.......X.] duration: 7.425573423s FAIL (2025-08-14 23:10:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 14 23:09:30.496311157 (149h43m24.927391288s) [upload=155721]: Starting upload Aug 14 23:09:30.496358919 (149h43m24.92743905s) [upload=155721]: Done starting upload Aug 14 23:09:30.496409976 (149h43m24.927490107s) [upload=155721]: Uploading segments... Aug 14 23:09:30.496446436 (149h43m24.927526567s) [upload=155721]: Got next segment. Inline: true Aug 14 23:09:30.496450264 (149h43m24.927530395s) [upload=155721]: Waiting for error group managing segments... Aug 14 23:09:30.496483657 (149h43m24.927563788s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:09:30.496487354 (149h43m24.927567485s) [upload=155721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:09:30.496489648 (149h43m24.927569779s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:09:33.939233724 (149h43m28.370313845s) [upload=155721]: Done uploading segments... Aug 14 23:09:33.939252460 (149h43m28.370332591s) [upload=155721]: Upload finished. err: <nil> Duration:* 7.425629912s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.288628397s FAIL (2025-08-14 23:09:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.288652783s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................................X.] duration: 6.054791503s FAIL (2025-08-14 23:09:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 14 23:08:35.105900497 (149h42m29.536980628s) [upload=155706]: Starting upload Aug 14 23:08:35.105956424 (149h42m29.537036555s) [upload=155706]: Done starting upload Aug 14 23:08:35.106058068 (149h42m29.537138199s) [upload=155706]: Uploading segments... Aug 14 23:08:35.106266597 (149h42m29.537346728s) [upload=155706]: Got next segment. Inline: false Aug 14 23:08:35.106292647 (149h42m29.537372778s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:08:35.106296905 (149h42m29.537377036s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 23:08:35.664735283 (149h42m30.095815414s) [seg_pos={0 0}, upload=155706]: Begin upload segment... Aug 14 23:08:35.664826968 (149h42m30.095907099s) [seg_pos={0 0}, upload=155706]: Done begin upload segment. Aug 14 23:08:35.664850333 (149h42m30.095930454s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.664953391 (149h42m30.096033522s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665339700 (149h42m30.096419831s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665585180 (149h42m30.096665311s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665729355 (149h42m30.096809486s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666053666 (149h42m30.097133797s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666335725 (149h42m30.097415896s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666464402 (149h42m30.097544533s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666808389 (149h42m30.09788852s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666990067 (149h42m30.098070208s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667066173 (149h42m30.098146304s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667286054 (149h42m30.098366185s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667410852 (149h42m30.098490983s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667548656 (149h42m30.098628787s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667718280 (149h42m30.098798401s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668161589 (149h42m30.09924171s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668464248 (149h42m30.099544379s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668807865 (149h42m30.099887996s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:36.779607273 (149h42m31.210687404s) [upload=155706]: Next returned nil segment Aug 14 23:08:36.779883893 (149h42m31.210964014s) [upload=155706]: Waiting for error group managing segments... Aug 14 23:08:37.080252078 (149h42m31.511332209s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.163526031 (149h42m31.594606162s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.201950862 (149h42m31.633030993s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.655662129 (149h42m32.08674226s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.686468463 (149h42m32.117548594s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.714936814 (149h42m32.146016945s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.751955404 (149h42m32.183035525s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.766303347 (149h42m32.197383478s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.819588991 (149h42m32.250669123s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.865209390 (149h42m32.296289521s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.257586512 (149h42m32.688666663s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.438386389 (149h42m32.86946652s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.502515471 (149h42m32.933595593s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826762315 (149h42m33.257842446s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826789567 (149h42m33.257869698s) [seg_pos={0 0}, upload=155706]: Segment reached optimal threshold of 14 pieces. Aug 14 23:08:38.826996513 (149h42m33.258076644s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: rpc: dial tcp 104.36.82.46:4502: operation was canceled; conn closed: op Aug 14 23:08:38.827180856 (149h42m33.258260977s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms): protocol: read tcp 188.166.225.170:35194->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827248545 (149h42m33.258328666s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection Aug 14 23:08:38.827256730 (149h42m33.258336862s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: read tcp 188.166.225.170:38054->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827279524 (149h42m33.258359655s) [seg_pos={0 0}, upload=155706]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 23:08:38.827354087 (149h42m33.258434218s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 23:08:38.827361080 (149h42m33.258441211s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:08:39.769810673 (149h42m34.200890804s) [upload=155706]: Done uploading segments... Aug 14 23:08:39.769827726 (149h42m34.200907857s) [upload=155706]: Upload finished. err: <nil> Duration: 6.054824708s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 6.259877528s FAIL (2025-08-14 23:09:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.259900002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [X......X.......X...................................X.......X] duration: 4.645734409s FAIL (2025-08-14 23:09:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 14 23:09:30.496311157 (149h43m24.927391288s) [upload=155721]: Starting upload Aug 14 23:09:30.496358919 (149h43m24.92743905s) [upload=155721]: Done starting upload Aug 14 23:09:30.496409976 (149h43m24.927490107s) [upload=155721]: Uploading segments... Aug 14 23:09:30.496446436 (149h43m24.927526567s) [upload=155721]: Got next segment. Inline: true Aug 14 23:09:30.496450264 (149h43m24.927530395s) [upload=155721]: Waiting for error group managing segments... Aug 14 23:09:30.496483657 (149h43m24.927563788s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:09:30.496487354 (149h43m24.927567485s) [upload=155721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:09:30.496489648 (149h43m24.927569779s) [upload=155721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:09:33.939233724 (149h43m28.370313845s) [upload=155721]: Done uploading segments... Aug 14 23:09:33.939252460 (149h43m28.370332591s) [upload=155721]: Upload finished. err: <nil> Duration:* 4.645785837s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 742.013746ms FAIL (2025-08-14 23:09:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 742.03758ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 722.843774ms FAIL (2025-08-14 23:09:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 722.863112ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.322231605s FAIL (2025-08-14 23:09:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.322254359s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X......X.......X...................................X.......] duration: 3.303477658s FAIL (2025-08-14 23:08:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 14 23:03:11.394691508 (149h37m5.825771659s) [upload=155615]: Starting upload Aug 14 23:03:11.394782743 (149h37m5.825862874s) [upload=155615]: Done starting upload Aug 14 23:03:11.394845162 (149h37m5.825925293s) [upload=155615]: Uploading segments... Aug 14 23:03:11.394880600 (149h37m5.825960731s) [upload=155615]: Got next segment. Inline: true Aug 14 23:03:11.394885931 (149h37m5.825966062s) [upload=155615]: Waiting for error group managing segments... Aug 14 23:03:11.394918012 (149h37m5.825998143s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:03:11.394923482 (149h37m5.826003613s) [upload=155615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:03:11.394925917 (149h37m5.826006048s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:03:11.801868054 (149h37m6.232948175s) [upload=155615]: Done uploading segments... Aug 14 23:03:11.801886991 (149h37m6.232967122s) [upload=155615]: Upload finished. err: <nil> Duration:* 3.303508506s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 6.441171904s FAIL (2025-08-14 23:08:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.441198425s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 6.608625035s FAIL (2025-08-14 23:08:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 14 23:08:35.105900497 (149h42m29.536980628s) [upload=155706]: Starting upload Aug 14 23:08:35.105956424 (149h42m29.537036555s) [upload=155706]: Done starting upload Aug 14 23:08:35.106058068 (149h42m29.537138199s) [upload=155706]: Uploading segments... Aug 14 23:08:35.106266597 (149h42m29.537346728s) [upload=155706]: Got next segment. Inline: false Aug 14 23:08:35.106292647 (149h42m29.537372778s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:08:35.106296905 (149h42m29.537377036s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 23:08:35.664735283 (149h42m30.095815414s) [seg_pos={0 0}, upload=155706]: Begin upload segment... Aug 14 23:08:35.664826968 (149h42m30.095907099s) [seg_pos={0 0}, upload=155706]: Done begin upload segment. Aug 14 23:08:35.664850333 (149h42m30.095930454s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.664953391 (149h42m30.096033522s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665339700 (149h42m30.096419831s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665585180 (149h42m30.096665311s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.665729355 (149h42m30.096809486s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666053666 (149h42m30.097133797s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666335725 (149h42m30.097415896s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666464402 (149h42m30.097544533s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666808389 (149h42m30.09788852s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.666990067 (149h42m30.098070208s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667066173 (149h42m30.098146304s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667286054 (149h42m30.098366185s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667410852 (149h42m30.098490983s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667548656 (149h42m30.098628787s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.667718280 (149h42m30.098798401s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668161589 (149h42m30.09924171s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668464248 (149h42m30.099544379s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:35.668807865 (149h42m30.099887996s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Uploading piece... Aug 14 23:08:36.779607273 (149h42m31.210687404s) [upload=155706]: Next returned nil segment Aug 14 23:08:36.779883893 (149h42m31.210964014s) [upload=155706]: Waiting for error group managing segments... Aug 14 23:08:37.080252078 (149h42m31.511332209s) [noise=true , address=64.49.13.137:4502 , piece_id=QEV7LN2X27EAXJ4LX2LAMHIZCQXHLQM56GJ3GZUEM2EQFEBL2ELA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.163526031 (149h42m31.594606162s) [noise=true , address=64.49.13.138:4481 , piece_id=WGPD24N3NF6MJX2BIPLMVKUBIQWZV22VQKB5353NN4TZ4K6M3GWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.201950862 (149h42m31.633030993s) [noise=true , address=64.49.13.26:4955 , piece_id=M66QFT76NCDV5KKZQC475YO7RXNCH66KWT47T3ETKGZWAK24WRWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.655662129 (149h42m32.08674226s) [noise=true , address=104.36.82.39:4756 , piece_id=2RHIQZ2E4CQCDG257KHAHBIG66YG6ML7KEABOD7TJ26ORCZMY4TA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.686468463 (149h42m32.117548594s) [noise=true , address=104.36.82.9:4018 , piece_id=P2KCG74VREIAVVRNIPZIV62R4T5Q3ZZYVEC7SPWK2NNIEH4ATTBA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.714936814 (149h42m32.146016945s) [noise=true , address=104.36.82.2:4752 , piece_id=DRL3TK2YATSVO5J3NUIBQAB3VMPXM4EL5A4F7EHKC7WGE7NTQ7GQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.751955404 (149h42m32.183035525s) [noise=true , address=104.36.82.12:4168 , piece_id=TRIDQGENHNLJPLV2TACKLVXDBIKDISUVJVQIRQTCUBZZPQABWEBQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.766303347 (149h42m32.197383478s) [noise=true , address=104.36.82.59:4313 , piece_id=OEJ2SXNUPQSKQJRSE7I44PT722UJBQGLOSHEFGPTJWKKGPFQ5JPA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.819588991 (149h42m32.250669123s) [noise=true , address=104.36.82.61:4375 , piece_id=BFQM2KYUBFZ73TAOPCLJMTTT7ZBM6CVOAHH4SVWHB3B3Y7R2ZWTQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:37.865209390 (149h42m32.296289521s) [noise=true , address=104.36.82.7:4924 , piece_id=KJ5GNEWEYILLT4UO5IKBXL5GP4SRNZGIGEJRTWJXWYVC3NLZCS2Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.257586512 (149h42m32.688666663s) [noise=true , address=77.90.31.12:4619 , piece_id=I4J5AN2WN4JYDKKWZSKTTBHD47T444BLGLNLQGPOTNZOOSAQLVZQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.438386389 (149h42m32.86946652s) [noise=true , address=77.90.31.16:4741 , piece_id=3LFTRJQWLK2PVXJDYNCRXQHO277NMDGSRMGWBDA4IDNUREHH7AXA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.502515471 (149h42m32.933595593s) [noise=true , address=104.36.82.36:4629 , piece_id=DN6UCBRRT6UXCHHPY2IFL4ICNVK7VHVOVEQMNKLMHDLKYFEX66PA, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826762315 (149h42m33.257842446s) [noise=true , address=104.36.82.56:4248 , piece_id=E6YV7JCHRHNJL3GIIUBDXPFL2AFJN7LSOIYLTUQLCNBV7QGQUKWQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: <nil> Aug 14 23:08:38.826789567 (149h42m33.257869698s) [seg_pos={0 0}, upload=155706]: Segment reached optimal threshold of 14 pieces. Aug 14 23:08:38.826996513 (149h42m33.258076644s) [noise=true , address=104.36.82.46:4502 , piece_id=GV63LSUBZIHZ7R2OZN6SSENPBQDPIDOMC2KR4VJOOIN7F2JIXKYQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: rpc: dial tcp 104.36.82.46:4502: operation was canceled; conn closed: op Aug 14 23:08:38.827180856 (149h42m33.258260977s) [noise=true , address=104.36.82.23:4450 , piece_id=FP7MMCBHVTMV3F5BXCWT7IDYKWEOOG3Q7JOXDZGEIHKT4E5YRP4Q, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms): protocol: read tcp 188.166.225.170:35194->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827248545 (149h42m33.258328666s) [noise=true , address=104.36.82.37:4681 , piece_id=W3LQO4O6A74Y3I7E4OAQX3V5GGPAWFMZ3NHQH3T4HQYKTXYGCWMQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48136->104.36.82.37:4681: use of closed network connection Aug 14 23:08:38.827256730 (149h42m33.258336862s) [noise=true , address=104.36.82.44:4899 , piece_id=DEJAXSAXUW3BUNM62TPOPWTXKG7ZAOUPNJV5YUBLJQPAC42TFMJQ, seg_pos={0 0}, upload=155706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: read tcp 188.166.225.170:38054->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 23:08:38.827279524 (149h42m33.258359655s) [seg_pos={0 0}, upload=155706]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 23:08:38.827354087 (149h42m33.258434218s) [upload=155706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 23:08:38.827361080 (149h42m33.258441211s) [upload=155706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:08:39.769810673 (149h42m34.200890804s) [upload=155706]: Done uploading segments... Aug 14 23:08:39.769827726 (149h42m34.200907857s) [upload=155706]: Upload finished. err: <nil> Duration: 6.608650973s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 752.795766ms FAIL (2025-08-14 23:08:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 752.815994ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.481620363s FAIL (2025-08-14 23:08:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.481641222s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 727.516707ms FAIL (2025-08-14 23:08:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 727.54407ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X......X.......X...................................X......] duration: 3.09816062s FAIL (2025-08-14 23:07:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 14 23:03:11.394691508 (149h37m5.825771659s) [upload=155615]: Starting upload Aug 14 23:03:11.394782743 (149h37m5.825862874s) [upload=155615]: Done starting upload Aug 14 23:03:11.394845162 (149h37m5.825925293s) [upload=155615]: Uploading segments... Aug 14 23:03:11.394880600 (149h37m5.825960731s) [upload=155615]: Got next segment. Inline: true Aug 14 23:03:11.394885931 (149h37m5.825966062s) [upload=155615]: Waiting for error group managing segments... Aug 14 23:03:11.394918012 (149h37m5.825998143s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:03:11.394923482 (149h37m5.826003613s) [upload=155615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:03:11.394925917 (149h37m5.826006048s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:03:11.801868054 (149h37m6.232948175s) [upload=155615]: Done uploading segments... Aug 14 23:03:11.801886991 (149h37m6.232967122s) [upload=155615]: Upload finished. err: <nil> Duration:* 3.098183304s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.652492594s FAIL (2025-08-14 23:07:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 584456c42d708087c09f0faeed3d1694451ccddbcd31c02942c9bab22782a5ee != before 5c56d2ec81ba59faf7cfbab91a8ddcef2d8a400fc52930931b158cbc222317fa ======LOGS====== Aug 14 22:26:13.497682333 (149h0m7.928762464s) [upload=154981]: Starting upload Aug 14 22:26:13.497731406 (149h0m7.928811537s) [upload=154981]: Done starting upload Aug 14 22:26:13.497858279 (149h0m7.92893842s) [upload=154981]: Uploading segments... Aug 14 22:26:13.498075795 (149h0m7.929155916s) [upload=154981]: Got next segment. Inline: false Aug 14 22:26:13.498128997 (149h0m7.929209118s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 22:26:13.498133646 (149h0m7.929213777s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 22:26:13.992314623 (149h0m8.423394744s) [seg_pos={0 0}, upload=154981]: Begin upload segment... Aug 14 22:26:13.992365701 (149h0m8.423445832s) [seg_pos={0 0}, upload=154981]: Done begin upload segment. Aug 14 22:26:13.992387002 (149h0m8.423467133s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992454140 (149h0m8.423534271s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992940611 (149h0m8.424020742s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.993072905 (149h0m8.424153036s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994062568 (149h0m8.425142699s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994133022 (149h0m8.425213153s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994388251 (149h0m8.425468383s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994595357 (149h0m8.425675488s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994800130 (149h0m8.425880261s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995007416 (149h0m8.426087547s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995221876 (149h0m8.426302007s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995422941 (149h0m8.426503073s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995617393 (149h0m8.426697525s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995813909 (149h0m8.42689404s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996000406 (149h0m8.427080537s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996185730 (149h0m8.427265871s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.998605130 (149h0m8.429685251s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.999270543 (149h0m8.430350664s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:15.479210578 (149h0m9.910290699s) [upload=154981]: Next returned nil segment Aug 14 22:26:15.479224896 (149h0m9.910305027s) [upload=154981]: Waiting for error group managing segments... Aug 14 22:26:15.884191582 (149h0m10.315271713s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:15.979809937 (149h0m10.410890068s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.038643598 (149h0m10.469723719s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.046584533 (149h0m10.477664664s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.060061459 (149h0m10.49114159s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.109095179 (149h0m10.54017531s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.125159296 (149h0m10.556239427s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.127764371 (149h0m10.558844502s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.274353653 (149h0m10.705433784s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.432231163 (149h0m10.863311294s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.479539401 (149h0m10.910619532s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.517647347 (149h0m10.948727468s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.698196867 (149h0m11.129276988s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701477545 (149h0m11.132557706s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701494006 (149h0m11.132574137s) [seg_pos={0 0}, upload=154981]: Segment reached optimal threshold of 14 pieces. Aug 14 22:26:16.701792186 (149h0m11.132872327s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6): protocol: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection Aug 14 22:26:16.701887078 (149h0m11.132967209s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM): protocol: read tcp 188.166.225.170:38298->104.36.82.3:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702037776 (149h0m11.133117957s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3): protocol: read tcp 188.166.225.170:43306->104.36.82.3:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702046854 (149h0m11.133126985s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HX5YAgmkBYJyw6bAY1vz4ov5aBeDsXjU4s7pxpBKAyLbG8yN6): protocol: read tcp 188.166.225.170:52080->104.36.82.20:4452: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702087111 (149h0m11.133167242s) [seg_pos={0 0}, upload=154981]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 22:26:16.702169368 (149h0m11.133249499s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 22:26:16.702173857 (149h0m11.133253988s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 22:26:17.128005086 (149h0m11.559085237s) [upload=154981]: Done uploading segments... Aug 14 22:26:17.128028250 (149h0m11.559108401s) [upload=154981]: Upload finished. err: <nil> Duration: 6.652518063s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 6.06432302s FAIL (2025-08-14 23:07:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.064345283s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.368740361s FAIL (2025-08-14 23:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.368765357s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...X......X.......X...................................X.....] duration: 2.363893219s FAIL (2025-08-14 23:07:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 14 23:03:11.394691508 (149h37m5.825771659s) [upload=155615]: Starting upload Aug 14 23:03:11.394782743 (149h37m5.825862874s) [upload=155615]: Done starting upload Aug 14 23:03:11.394845162 (149h37m5.825925293s) [upload=155615]: Uploading segments... Aug 14 23:03:11.394880600 (149h37m5.825960731s) [upload=155615]: Got next segment. Inline: true Aug 14 23:03:11.394885931 (149h37m5.825966062s) [upload=155615]: Waiting for error group managing segments... Aug 14 23:03:11.394918012 (149h37m5.825998143s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:03:11.394923482 (149h37m5.826003613s) [upload=155615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:03:11.394925917 (149h37m5.826006048s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:03:11.801868054 (149h37m6.232948175s) [upload=155615]: Done uploading segments... Aug 14 23:03:11.801886991 (149h37m6.232967122s) [upload=155615]: Upload finished. err: <nil> Duration:* 2.363922846s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.297868377s FAIL (2025-08-14 23:06:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.297882224s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 778.804418ms FAIL (2025-08-14 23:06:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 778.830458ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.372718555s FAIL (2025-08-14 23:06:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 584456c42d708087c09f0faeed3d1694451ccddbcd31c02942c9bab22782a5ee != before 5c56d2ec81ba59faf7cfbab91a8ddcef2d8a400fc52930931b158cbc222317fa ======LOGS====== Aug 14 22:26:13.497682333 (149h0m7.928762464s) [upload=154981]: Starting upload Aug 14 22:26:13.497731406 (149h0m7.928811537s) [upload=154981]: Done starting upload Aug 14 22:26:13.497858279 (149h0m7.92893842s) [upload=154981]: Uploading segments... Aug 14 22:26:13.498075795 (149h0m7.929155916s) [upload=154981]: Got next segment. Inline: false Aug 14 22:26:13.498128997 (149h0m7.929209118s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 22:26:13.498133646 (149h0m7.929213777s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 22:26:13.992314623 (149h0m8.423394744s) [seg_pos={0 0}, upload=154981]: Begin upload segment... Aug 14 22:26:13.992365701 (149h0m8.423445832s) [seg_pos={0 0}, upload=154981]: Done begin upload segment. Aug 14 22:26:13.992387002 (149h0m8.423467133s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992454140 (149h0m8.423534271s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992940611 (149h0m8.424020742s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.993072905 (149h0m8.424153036s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994062568 (149h0m8.425142699s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994133022 (149h0m8.425213153s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994388251 (149h0m8.425468383s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994595357 (149h0m8.425675488s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994800130 (149h0m8.425880261s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995007416 (149h0m8.426087547s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995221876 (149h0m8.426302007s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995422941 (149h0m8.426503073s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995617393 (149h0m8.426697525s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995813909 (149h0m8.42689404s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996000406 (149h0m8.427080537s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996185730 (149h0m8.427265871s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.998605130 (149h0m8.429685251s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.999270543 (149h0m8.430350664s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:15.479210578 (149h0m9.910290699s) [upload=154981]: Next returned nil segment Aug 14 22:26:15.479224896 (149h0m9.910305027s) [upload=154981]: Waiting for error group managing segments... Aug 14 22:26:15.884191582 (149h0m10.315271713s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:15.979809937 (149h0m10.410890068s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.038643598 (149h0m10.469723719s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.046584533 (149h0m10.477664664s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.060061459 (149h0m10.49114159s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.109095179 (149h0m10.54017531s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.125159296 (149h0m10.556239427s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.127764371 (149h0m10.558844502s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.274353653 (149h0m10.705433784s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.432231163 (149h0m10.863311294s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.479539401 (149h0m10.910619532s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.517647347 (149h0m10.948727468s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.698196867 (149h0m11.129276988s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701477545 (149h0m11.132557706s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701494006 (149h0m11.132574137s) [seg_pos={0 0}, upload=154981]: Segment reached optimal threshold of 14 pieces. Aug 14 22:26:16.701792186 (149h0m11.132872327s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6): protocol: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection Aug 14 22:26:16.701887078 (149h0m11.132967209s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM): protocol: read tcp 188.166.225.170:38298->104.36.82.3:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702037776 (149h0m11.133117957s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3): protocol: read tcp 188.166.225.170:43306->104.36.82.3:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702046854 (149h0m11.133126985s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HX5YAgmkBYJyw6bAY1vz4ov5aBeDsXjU4s7pxpBKAyLbG8yN6): protocol: read tcp 188.166.225.170:52080->104.36.82.20:4452: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702087111 (149h0m11.133167242s) [seg_pos={0 0}, upload=154981]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 22:26:16.702169368 (149h0m11.133249499s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 22:26:16.702173857 (149h0m11.133253988s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 22:26:17.128005086 (149h0m11.559085237s) [upload=154981]: Done uploading segments... Aug 14 22:26:17.128028250 (149h0m11.559108401s) [upload=154981]: Upload finished. err: <nil> Duration: 6.372748674s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....X......X.......X...................................X....] duration: 1.883791115s FAIL (2025-08-14 23:06:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 14 23:03:11.394691508 (149h37m5.825771659s) [upload=155615]: Starting upload Aug 14 23:03:11.394782743 (149h37m5.825862874s) [upload=155615]: Done starting upload Aug 14 23:03:11.394845162 (149h37m5.825925293s) [upload=155615]: Uploading segments... Aug 14 23:03:11.394880600 (149h37m5.825960731s) [upload=155615]: Got next segment. Inline: true Aug 14 23:03:11.394885931 (149h37m5.825966062s) [upload=155615]: Waiting for error group managing segments... Aug 14 23:03:11.394918012 (149h37m5.825998143s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:03:11.394923482 (149h37m5.826003613s) [upload=155615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:03:11.394925917 (149h37m5.826006048s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:03:11.801868054 (149h37m6.232948175s) [upload=155615]: Done uploading segments... Aug 14 23:03:11.801886991 (149h37m6.232967122s) [upload=155615]: Upload finished. err: <nil> Duration:* 1.883813939s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 721.003927ms FAIL (2025-08-14 23:06:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 721.029746ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.XXXXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 6.287370744s FAIL (2025-08-14 23:06:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.28739532s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.285516372s FAIL (2025-08-14 23:06:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.285543735s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.301630969s FAIL (2025-08-14 23:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 584456c42d708087c09f0faeed3d1694451ccddbcd31c02942c9bab22782a5ee != before 5c56d2ec81ba59faf7cfbab91a8ddcef2d8a400fc52930931b158cbc222317fa ======LOGS====== Aug 14 22:26:13.497682333 (149h0m7.928762464s) [upload=154981]: Starting upload Aug 14 22:26:13.497731406 (149h0m7.928811537s) [upload=154981]: Done starting upload Aug 14 22:26:13.497858279 (149h0m7.92893842s) [upload=154981]: Uploading segments... Aug 14 22:26:13.498075795 (149h0m7.929155916s) [upload=154981]: Got next segment. Inline: false Aug 14 22:26:13.498128997 (149h0m7.929209118s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 22:26:13.498133646 (149h0m7.929213777s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 22:26:13.992314623 (149h0m8.423394744s) [seg_pos={0 0}, upload=154981]: Begin upload segment... Aug 14 22:26:13.992365701 (149h0m8.423445832s) [seg_pos={0 0}, upload=154981]: Done begin upload segment. Aug 14 22:26:13.992387002 (149h0m8.423467133s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992454140 (149h0m8.423534271s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992940611 (149h0m8.424020742s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.993072905 (149h0m8.424153036s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994062568 (149h0m8.425142699s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994133022 (149h0m8.425213153s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994388251 (149h0m8.425468383s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994595357 (149h0m8.425675488s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994800130 (149h0m8.425880261s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995007416 (149h0m8.426087547s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995221876 (149h0m8.426302007s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995422941 (149h0m8.426503073s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995617393 (149h0m8.426697525s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995813909 (149h0m8.42689404s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996000406 (149h0m8.427080537s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996185730 (149h0m8.427265871s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.998605130 (149h0m8.429685251s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.999270543 (149h0m8.430350664s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:15.479210578 (149h0m9.910290699s) [upload=154981]: Next returned nil segment Aug 14 22:26:15.479224896 (149h0m9.910305027s) [upload=154981]: Waiting for error group managing segments... Aug 14 22:26:15.884191582 (149h0m10.315271713s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:15.979809937 (149h0m10.410890068s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.038643598 (149h0m10.469723719s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.046584533 (149h0m10.477664664s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.060061459 (149h0m10.49114159s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.109095179 (149h0m10.54017531s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.125159296 (149h0m10.556239427s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.127764371 (149h0m10.558844502s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.274353653 (149h0m10.705433784s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.432231163 (149h0m10.863311294s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.479539401 (149h0m10.910619532s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.517647347 (149h0m10.948727468s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.698196867 (149h0m11.129276988s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701477545 (149h0m11.132557706s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701494006 (149h0m11.132574137s) [seg_pos={0 0}, upload=154981]: Segment reached optimal threshold of 14 pieces. Aug 14 22:26:16.701792186 (149h0m11.132872327s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6): protocol: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection Aug 14 22:26:16.701887078 (149h0m11.132967209s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM): protocol: read tcp 188.166.225.170:38298->104.36.82.3:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702037776 (149h0m11.133117957s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3): protocol: read tcp 188.166.225.170:43306->104.36.82.3:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702046854 (149h0m11.133126985s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HX5YAgmkBYJyw6bAY1vz4ov5aBeDsXjU4s7pxpBKAyLbG8yN6): protocol: read tcp 188.166.225.170:52080->104.36.82.20:4452: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702087111 (149h0m11.133167242s) [seg_pos={0 0}, upload=154981]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 22:26:16.702169368 (149h0m11.133249499s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 22:26:16.702173857 (149h0m11.133253988s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 22:26:17.128005086 (149h0m11.559085237s) [upload=154981]: Done uploading segments... Aug 14 22:26:17.128028250 (149h0m11.559108401s) [upload=154981]: Upload finished. err: <nil> Duration: 6.301654644s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.358225013s FAIL (2025-08-14 23:05:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.358244109s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [..XXXXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 6.363246265s FAIL (2025-08-14 23:05:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.363313604s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 739.195516ms FAIL (2025-08-14 23:05:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 739.218391ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X......X.......X...................................X...] duration: 1.87188651s FAIL (2025-08-14 23:05:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 14 23:03:11.394691508 (149h37m5.825771659s) [upload=155615]: Starting upload Aug 14 23:03:11.394782743 (149h37m5.825862874s) [upload=155615]: Done starting upload Aug 14 23:03:11.394845162 (149h37m5.825925293s) [upload=155615]: Uploading segments... Aug 14 23:03:11.394880600 (149h37m5.825960731s) [upload=155615]: Got next segment. Inline: true Aug 14 23:03:11.394885931 (149h37m5.825966062s) [upload=155615]: Waiting for error group managing segments... Aug 14 23:03:11.394918012 (149h37m5.825998143s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:03:11.394923482 (149h37m5.826003613s) [upload=155615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:03:11.394925917 (149h37m5.826006048s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:03:11.801868054 (149h37m6.232948175s) [upload=155615]: Done uploading segments... Aug 14 23:03:11.801886991 (149h37m6.232967122s) [upload=155615]: Upload finished. err: <nil> Duration:* 1.871914834s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.283324772s FAIL (2025-08-14 23:05:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 584456c42d708087c09f0faeed3d1694451ccddbcd31c02942c9bab22782a5ee != before 5c56d2ec81ba59faf7cfbab91a8ddcef2d8a400fc52930931b158cbc222317fa ======LOGS====== Aug 14 22:26:13.497682333 (149h0m7.928762464s) [upload=154981]: Starting upload Aug 14 22:26:13.497731406 (149h0m7.928811537s) [upload=154981]: Done starting upload Aug 14 22:26:13.497858279 (149h0m7.92893842s) [upload=154981]: Uploading segments... Aug 14 22:26:13.498075795 (149h0m7.929155916s) [upload=154981]: Got next segment. Inline: false Aug 14 22:26:13.498128997 (149h0m7.929209118s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 22:26:13.498133646 (149h0m7.929213777s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 14 22:26:13.992314623 (149h0m8.423394744s) [seg_pos={0 0}, upload=154981]: Begin upload segment... Aug 14 22:26:13.992365701 (149h0m8.423445832s) [seg_pos={0 0}, upload=154981]: Done begin upload segment. Aug 14 22:26:13.992387002 (149h0m8.423467133s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992454140 (149h0m8.423534271s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.992940611 (149h0m8.424020742s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.993072905 (149h0m8.424153036s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994062568 (149h0m8.425142699s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994133022 (149h0m8.425213153s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994388251 (149h0m8.425468383s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994595357 (149h0m8.425675488s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.994800130 (149h0m8.425880261s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995007416 (149h0m8.426087547s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995221876 (149h0m8.426302007s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995422941 (149h0m8.426503073s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995617393 (149h0m8.426697525s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.995813909 (149h0m8.42689404s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996000406 (149h0m8.427080537s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.996185730 (149h0m8.427265871s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.998605130 (149h0m8.429685251s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:13.999270543 (149h0m8.430350664s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Uploading piece... Aug 14 22:26:15.479210578 (149h0m9.910290699s) [upload=154981]: Next returned nil segment Aug 14 22:26:15.479224896 (149h0m9.910305027s) [upload=154981]: Waiting for error group managing segments... Aug 14 22:26:15.884191582 (149h0m10.315271713s) [noise=true , address=79.127.220.101:4062 , piece_id=EB3WGWWNK3WC5L7QBAXINF45X3QG2UBXKC7NQDIMG2MC37W574IQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:15.979809937 (149h0m10.410890068s) [noise=true , address=104.36.82.22:4519 , piece_id=TRBOMMTO2D6H7IZOIZG7CJ4L7WW7JI67GJCZPEH7GZGOUO3JLT4A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.038643598 (149h0m10.469723719s) [noise=true , address=104.36.82.23:4459 , piece_id=R3VTPCYRCJE5BAQJKDG76UBIZW3WQADZIRPPPVLORFHAPPANQ6RQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.046584533 (149h0m10.477664664s) [noise=true , address=79.127.201.215:4343 , piece_id=32PBVWDURI6ZKXXVR5JUTRFOFZO63EWKAW5ZWKMJAKCSZVTMKPMQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.060061459 (149h0m10.49114159s) [noise=true , address=77.90.31.15:4721 , piece_id=ZDMZ772CRI3JJKIURGTAWR3GERAXUT5PPJ367KMFTGFESLEFWN5Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.109095179 (149h0m10.54017531s) [noise=true , address=104.36.82.2:4751 , piece_id=AN26TMRGCIDRWZJLRZ3D2RAPKGFWYG2SCPMVQIDDBGSULPR3CN3Q, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.125159296 (149h0m10.556239427s) [noise=true , address=104.36.82.5:4868 , piece_id=5IQVXHIE6U5OU7HX7AOKCZCNTAZTSY6OXULF2EIKNLNVEOYSPYCA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.127764371 (149h0m10.558844502s) [noise=true , address=104.36.82.58:4274 , piece_id=3O35Z7OA4PHQELTWQMOWZBU4OYOEGU6PTAV3UOJIRGCEZBNFEGEA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.274353653 (149h0m10.705433784s) [noise=true , address=77.90.31.21:4904 , piece_id=25ESPAPI5EX3PEK5DOKSGQJUY67XA7T5ABRA6KWXUEEYB2HRDITQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.432231163 (149h0m10.863311294s) [noise=true , address=77.90.31.13:4631 , piece_id=4V6PL6A2LX4LU4FX3FQKAIN75TMNSBPLAFIKESAKNGRZQWH2FCMA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.479539401 (149h0m10.910619532s) [noise=true , address=77.90.31.16:4749 , piece_id=YH4S54NRP375O66L7SRBMK3QPGXFUVREEBJOG55JHZALMZBVEYWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.517647347 (149h0m10.948727468s) [noise=true , address=77.90.31.12:4606 , piece_id=JAUXFZRHY3XNV3IU3FGMEDT6ILKAHV6Z7P5EPP4NYFXSVP22CAWA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.698196867 (149h0m11.129276988s) [noise=true , address=104.36.82.60:4359 , piece_id=6AN4XA4S54IBZZOISGGKUIIJXCAEDCOX3UDZBMSMSL47LAMX4VUQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701477545 (149h0m11.132557706s) [noise=true , address=104.36.82.45:4937 , piece_id=N5WKBIV2L5LFW3MWMCHTHQEMXFWK3GS4KVGZ6ZCBYZ57WWFZJUPQ, seg_pos={0 0}, upload=154981]: Done uploading piece. err: <nil> Aug 14 22:26:16.701494006 (149h0m11.132574137s) [seg_pos={0 0}, upload=154981]: Segment reached optimal threshold of 14 pieces. Aug 14 22:26:16.701792186 (149h0m11.132872327s) [noise=true , address=104.36.82.47:4527 , piece_id=AKTBF5SBOIOFPC6XWQ6RIBAUBCBBCPNJFNW7YLGV3IW573EGUOZA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6): protocol: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->104.36.82.47:4527: use of closed network connection Aug 14 22:26:16.701887078 (149h0m11.132967209s) [noise=true , address=104.36.82.3:4780 , piece_id=JXS3SG57OPOQNGKXKFSOOF4XAH2GNE4SDN2OG2WVZYI6PCC5YGYA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM): protocol: read tcp 188.166.225.170:38298->104.36.82.3:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702037776 (149h0m11.133117957s) [noise=true , address=104.36.82.3:4784 , piece_id=S2G2YJNQB54DYHVD6JWNQ47PTD2BEYUSF4OLEA3IW2JQFKFBWA3A, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3): protocol: read tcp 188.166.225.170:43306->104.36.82.3:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702046854 (149h0m11.133126985s) [noise=true , address=104.36.82.20:4452 , piece_id=KAKFHVV6DYHJGVEKEXXNHINQRWO3WWWRVEGV5WRMX5TW6KFCMBNA, seg_pos={0 0}, upload=154981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HX5YAgmkBYJyw6bAY1vz4ov5aBeDsXjU4s7pxpBKAyLbG8yN6): protocol: read tcp 188.166.225.170:52080->104.36.82.20:4452: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 14 22:26:16.702087111 (149h0m11.133167242s) [seg_pos={0 0}, upload=154981]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 14 22:26:16.702169368 (149h0m11.133249499s) [upload=154981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 14 22:26:16.702173857 (149h0m11.133253988s) [upload=154981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 22:26:17.128005086 (149h0m11.559085237s) [upload=154981]: Done uploading segments... Aug 14 22:26:17.128028250 (149h0m11.559108401s) [upload=154981]: Upload finished. err: <nil> Duration: 6.283351332s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [...XXXXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 6.32856048s FAIL (2025-08-14 23:05:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.328585098s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X......X.......X...................................X..] duration: 2.331066972s FAIL (2025-08-14 23:05:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 14 23:03:11.394691508 (149h37m5.825771659s) [upload=155615]: Starting upload Aug 14 23:03:11.394782743 (149h37m5.825862874s) [upload=155615]: Done starting upload Aug 14 23:03:11.394845162 (149h37m5.825925293s) [upload=155615]: Uploading segments... Aug 14 23:03:11.394880600 (149h37m5.825960731s) [upload=155615]: Got next segment. Inline: true Aug 14 23:03:11.394885931 (149h37m5.825966062s) [upload=155615]: Waiting for error group managing segments... Aug 14 23:03:11.394918012 (149h37m5.825998143s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 14 23:03:11.394923482 (149h37m5.826003613s) [upload=155615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 14 23:03:11.394925917 (149h37m5.826006048s) [upload=155615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 14 23:03:11.801868054 (149h37m6.232948175s) [upload=155615]: Done uploading segments... Aug 14 23:03:11.801886991 (149h37m6.232967122s) [upload=155615]: Upload finished. err: <nil> Duration:* 2.331097059s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [....XXXXXXXX.XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 6.394127422s FAIL (2025-08-14 23:04:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.394156568s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.532878854s FAIL (2025-08-14 23:04:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.532910997s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 767.248583ms FAIL (2025-08-14 23:04:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": remote error: tls: internal error Duration: 767.281075ms edge authservice http us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)