Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X........X.......................X........................] duration:
6.290626239s
FAIL (2025-07-09 20:02:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.290650986s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X........X.......................X.......................] duration:
6.172879658s
FAIL (2025-07-09 20:02:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.17290166s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................X..X.....................X...] duration:
10.623529762s
FAIL (2025-07-09 20:02:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315898]: Got next segment. Inline: false
Jul 9 20:00:01.267693251 (394h49m36.038334586s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 20:00:01.267700135 (394h49m36.03834147s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 20:00:01.715201952 (394h49m36.485843307s) [seg_pos={0 0}, upload=315898]: Begin upload segment...
Jul 9 20:00:01.715296183 (394h49m36.485937528s) [seg_pos={0 0}, upload=315898]: Done begin upload segment.
Jul 9 20:00:01.715332582 (394h49m36.485973937s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715353733 (394h49m36.485995078s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715413648 (394h49m36.486054993s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715839523 (394h49m36.486480868s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715854491 (394h49m36.486495836s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716252824 (394h49m36.486894169s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716274825 (394h49m36.48691617s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716770403 (394h49m36.487411768s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716807425 (394h49m36.48744877s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717175559 (394h49m36.487816904s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717261513 (394h49m36.487902858s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717562689 (394h49m36.488204034s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717649675 (394h49m36.488291041s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718081622 (394h49m36.488722967s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718456659 (394h49m36.489098004s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.719830938 (394h49m36.490472294s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.720421469 (394h49m36.491062864s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.721072935 (394h49m36.49171437s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:02.846673368 (394h49m37.617314713s) [upload=315898]: Next returned nil segment
Jul 9 20:00:02.846727541 (394h49m37.617368896s) [upload=315898]: Waiting for error group managing segments...
Jul 9 20:00:03.434946994 (394h49m38.205588339s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.629357310 (394h49m38.399998645s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.710145991 (394h49m38.480787336s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.724809153 (394h49m38.495450499s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.726766118 (394h49m38.497407463s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.736459613 (394h49m38.507100948s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.737328855 (394h49m38.5079702s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.744352673 (394h49m38.514994018s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.747871785 (394h49m38.518513131s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:07.148967050 (394h49m41.919608385s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:09.061157378 (394h49m43.831798713s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:13.841876001 (394h49m48.612517336s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:45.484870134 (394h50m20.255511469s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:01:00.622075620 (394h50m35.392716945s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:104.36.82.36:4631): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622243040 (394h50m35.392884375s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:15y4L1tWUv5vFniK5ugNHNYa22q7a7hC5Uf9pxkN4AuPBodcgY, address:104.36.82.20:4462): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622283868 (394h50m35.392925203s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: expected piece hash; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; EOF
Jul 9 20:01:00.622537793 (394h50m35.393179128s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:104.36.82.17:4332): protocol: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection
Jul 9 20:01:00.622584353 (394h50m35.393225688s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:104.36.82.34:4546): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622614701 (394h50m35.393256036s) [seg_pos={0 0}, upload=315898]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 20:01:00.859090035 (394h50m35.62973137s) [upload=315898]: Done uploading segments...
Jul 9 20:01:00.859115674 (394h50m35.629757009s) [upload=315898]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 10.62355961s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................X..X.....................X..] duration:
19.455972537s
FAIL (2025-07-09 20:01:49 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315898]: Got next segment. Inline: false
Jul 9 20:00:01.267693251 (394h49m36.038334586s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 20:00:01.267700135 (394h49m36.03834147s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 20:00:01.715201952 (394h49m36.485843307s) [seg_pos={0 0}, upload=315898]: Begin upload segment...
Jul 9 20:00:01.715296183 (394h49m36.485937528s) [seg_pos={0 0}, upload=315898]: Done begin upload segment.
Jul 9 20:00:01.715332582 (394h49m36.485973937s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715353733 (394h49m36.485995078s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715413648 (394h49m36.486054993s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715839523 (394h49m36.486480868s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715854491 (394h49m36.486495836s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716252824 (394h49m36.486894169s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716274825 (394h49m36.48691617s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716770403 (394h49m36.487411768s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716807425 (394h49m36.48744877s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717175559 (394h49m36.487816904s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717261513 (394h49m36.487902858s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717562689 (394h49m36.488204034s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717649675 (394h49m36.488291041s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718081622 (394h49m36.488722967s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718456659 (394h49m36.489098004s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.719830938 (394h49m36.490472294s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.720421469 (394h49m36.491062864s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.721072935 (394h49m36.49171437s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:02.846673368 (394h49m37.617314713s) [upload=315898]: Next returned nil segment
Jul 9 20:00:02.846727541 (394h49m37.617368896s) [upload=315898]: Waiting for error group managing segments...
Jul 9 20:00:03.434946994 (394h49m38.205588339s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.629357310 (394h49m38.399998645s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.710145991 (394h49m38.480787336s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.724809153 (394h49m38.495450499s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.726766118 (394h49m38.497407463s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.736459613 (394h49m38.507100948s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.737328855 (394h49m38.5079702s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.744352673 (394h49m38.514994018s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.747871785 (394h49m38.518513131s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:07.148967050 (394h49m41.919608385s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:09.061157378 (394h49m43.831798713s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:13.841876001 (394h49m48.612517336s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:45.484870134 (394h50m20.255511469s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:01:00.622075620 (394h50m35.392716945s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:104.36.82.36:4631): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622243040 (394h50m35.392884375s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:15y4L1tWUv5vFniK5ugNHNYa22q7a7hC5Uf9pxkN4AuPBodcgY, address:104.36.82.20:4462): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622283868 (394h50m35.392925203s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: expected piece hash; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; EOF
Jul 9 20:01:00.622537793 (394h50m35.393179128s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:104.36.82.17:4332): protocol: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection
Jul 9 20:01:00.622584353 (394h50m35.393225688s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:104.36.82.34:4546): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622614701 (394h50m35.393256036s) [seg_pos={0 0}, upload=315898]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 20:01:00.859090035 (394h50m35.62973137s) [upload=315898]: Done uploading segments...
Jul 9 20:01:00.859115674 (394h50m35.629757009s) [upload=315898]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 19.455990962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X.........X..X...X...] duration:
23.322246158s
FAIL (2025-07-09 20:01:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.322285594s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X........X.......................X......................] duration:
6.762320478s
FAIL (2025-07-09 20:01:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.762344594s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................X..X.....................X.] duration:
28.537369929s
FAIL (2025-07-09 20:01:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315898]: Got next segment. Inline: false
Jul 9 20:00:01.267693251 (394h49m36.038334586s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 20:00:01.267700135 (394h49m36.03834147s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 20:00:01.715201952 (394h49m36.485843307s) [seg_pos={0 0}, upload=315898]: Begin upload segment...
Jul 9 20:00:01.715296183 (394h49m36.485937528s) [seg_pos={0 0}, upload=315898]: Done begin upload segment.
Jul 9 20:00:01.715332582 (394h49m36.485973937s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715353733 (394h49m36.485995078s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715413648 (394h49m36.486054993s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715839523 (394h49m36.486480868s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715854491 (394h49m36.486495836s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716252824 (394h49m36.486894169s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716274825 (394h49m36.48691617s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716770403 (394h49m36.487411768s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716807425 (394h49m36.48744877s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717175559 (394h49m36.487816904s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717261513 (394h49m36.487902858s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717562689 (394h49m36.488204034s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717649675 (394h49m36.488291041s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718081622 (394h49m36.488722967s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718456659 (394h49m36.489098004s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.719830938 (394h49m36.490472294s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.720421469 (394h49m36.491062864s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.721072935 (394h49m36.49171437s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:02.846673368 (394h49m37.617314713s) [upload=315898]: Next returned nil segment
Jul 9 20:00:02.846727541 (394h49m37.617368896s) [upload=315898]: Waiting for error group managing segments...
Jul 9 20:00:03.434946994 (394h49m38.205588339s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.629357310 (394h49m38.399998645s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.710145991 (394h49m38.480787336s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.724809153 (394h49m38.495450499s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.726766118 (394h49m38.497407463s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.736459613 (394h49m38.507100948s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.737328855 (394h49m38.5079702s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.744352673 (394h49m38.514994018s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.747871785 (394h49m38.518513131s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:07.148967050 (394h49m41.919608385s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:09.061157378 (394h49m43.831798713s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:13.841876001 (394h49m48.612517336s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:45.484870134 (394h50m20.255511469s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:01:00.622075620 (394h50m35.392716945s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:104.36.82.36:4631): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622243040 (394h50m35.392884375s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:15y4L1tWUv5vFniK5ugNHNYa22q7a7hC5Uf9pxkN4AuPBodcgY, address:104.36.82.20:4462): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622283868 (394h50m35.392925203s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: expected piece hash; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; EOF
Jul 9 20:01:00.622537793 (394h50m35.393179128s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:104.36.82.17:4332): protocol: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection
Jul 9 20:01:00.622584353 (394h50m35.393225688s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:104.36.82.34:4546): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622614701 (394h50m35.393256036s) [seg_pos={0 0}, upload=315898]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 20:01:00.859090035 (394h50m35.62973137s) [upload=315898]: Done uploading segments...
Jul 9 20:01:00.859115674 (394h50m35.629757009s) [upload=315898]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 28.537389508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........................................X.........X..X...X..] duration:
53.662904797s
FAIL (2025-07-09 20:01:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 53.66292736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
4.83391143s
FAIL (2025-07-09 20:01:18 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.833935147s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................X..X.....................X] duration:
1m0.238222308s
FAIL (2025-07-09 20:01:00 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315898]: Got next segment. Inline: false
Jul 9 20:00:01.267693251 (394h49m36.038334586s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 20:00:01.267700135 (394h49m36.03834147s) [upload=315898]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 20:00:01.715201952 (394h49m36.485843307s) [seg_pos={0 0}, upload=315898]: Begin upload segment...
Jul 9 20:00:01.715296183 (394h49m36.485937528s) [seg_pos={0 0}, upload=315898]: Done begin upload segment.
Jul 9 20:00:01.715332582 (394h49m36.485973937s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715353733 (394h49m36.485995078s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715413648 (394h49m36.486054993s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715839523 (394h49m36.486480868s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.715854491 (394h49m36.486495836s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716252824 (394h49m36.486894169s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716274825 (394h49m36.48691617s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716770403 (394h49m36.487411768s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.716807425 (394h49m36.48744877s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717175559 (394h49m36.487816904s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717261513 (394h49m36.487902858s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717562689 (394h49m36.488204034s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.717649675 (394h49m36.488291041s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718081622 (394h49m36.488722967s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.718456659 (394h49m36.489098004s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.719830938 (394h49m36.490472294s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.720421469 (394h49m36.491062864s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:01.721072935 (394h49m36.49171437s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Uploading piece...
Jul 9 20:00:02.846673368 (394h49m37.617314713s) [upload=315898]: Next returned nil segment
Jul 9 20:00:02.846727541 (394h49m37.617368896s) [upload=315898]: Waiting for error group managing segments...
Jul 9 20:00:03.434946994 (394h49m38.205588339s) [noise=true , address=79.127.220.101:4062 , piece_id=SC2KCK7S4327ZJB3ILAUIL5Q2XWI2J4GBHFPQX7SRLOXLHPOE5AA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.629357310 (394h49m38.399998645s) [noise=true , address=79.127.201.215:4342 , piece_id=ITAJMMOKLXOC3F53AYANHCGR3EVVYJEA5KBFJRACFDZ6F3LSPEZA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.710145991 (394h49m38.480787336s) [noise=true , address=77.90.31.12:4611 , piece_id=3KC25O7DV2CKTIRPMORWUQQ4VERAOXR7XBWKW3IODAB5XOGMEQRA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.724809153 (394h49m38.495450499s) [noise=true , address=77.90.31.21:4902 , piece_id=RXRCF4IUXTPNSCBNRDSGGWBTZ4U4PJBGFQ73WZGTWURMWYHPTTMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.726766118 (394h49m38.497407463s) [noise=true , address=77.90.31.16:4739 , piece_id=VFIPXNATKRTP26LZG764GSCDXHGLOYXJ5NEDD5Q5PG6KJ2OJG2TQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.736459613 (394h49m38.507100948s) [noise=true , address=77.90.31.10:4547 , piece_id=VYBGPSHLJRHQO6I4YVBVUKLEOCOHKRDEJX73E42KMR34HFRAXRVQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.737328855 (394h49m38.5079702s) [noise=true , address=77.90.31.17:4769 , piece_id=GU7GSQUWVJU3OJRQO5L6RBIP7HEXVFG7XVOGXTE5W3COGQXIXFDA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.744352673 (394h49m38.514994018s) [noise=true , address=77.90.31.19:4857 , piece_id=FB6WLVLF5HNUD6ZIUIBAT76MDR25SWC6DGGSOYNMDVCDX4R52S7Q, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:03.747871785 (394h49m38.518513131s) [noise=true , address=77.90.31.15:4724 , piece_id=NC4ITOJ2TPWCDEQRG2UNFDATYDHSY6WEOAI6O7WSYT2DPKC75KKQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:07.148967050 (394h49m41.919608385s) [noise=true , address=79.127.201.215:4355 , piece_id=6WYNZ5YDULZ4IHUIQLZIQES5GMCXSXXTWIFK6RY4PSIGZX55IQMQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:09.061157378 (394h49m43.831798713s) [noise=true , address=79.127.219.47:4047 , piece_id=BSVCIZGYNCRVNPUP344AUGFZXYSIGJKMAU45ZYUWOFALNVDPA2PQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:13.841876001 (394h49m48.612517336s) [noise=true , address=79.127.219.38:4024 , piece_id=7QZOGDYRLNRPFYRHAU3FA52ZC4LUGHWMZSCGYOQ6P7KS5FOZ2IXA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:00:45.484870134 (394h50m20.255511469s) [noise=true , address=64.49.13.169:4706 , piece_id=UUWT3LNG4JE5JLSJKIEJVHC6H767QNRI4V4CE3WN6CT3HMTJ46MA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: <nil>
Jul 9 20:01:00.622075620 (394h50m35.392716945s) [noise=true , address=104.36.82.36:4631 , piece_id=VUQ25JOQ3HWQRFSMHELSZSUSLQGZEL2JEFJWD2D3SZHUJGW4H5FQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:104.36.82.36:4631): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622243040 (394h50m35.392884375s) [noise=true , address=104.36.82.20:4462 , piece_id=2MWIVCTHD4EJHGU6PGLEUDBDIMPAAQMC2S6XDRTEQAP3NLTL2BLA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:15y4L1tWUv5vFniK5ugNHNYa22q7a7hC5Uf9pxkN4AuPBodcgY, address:104.36.82.20:4462): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622283868 (394h50m35.392925203s) [noise=true , address=104.36.82.44:4873 , piece_id=JAJWWIKUN64LFQEFX57MND77Y7K67OD37U2HFIZ5U5VYLBAZQWLQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: expected piece hash; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; write tcp 188.166.225.170:37508->104.36.82.44:4873: use of closed network connection; EOF
Jul 9 20:01:00.622537793 (394h50m35.393179128s) [noise=true , address=104.36.82.17:4332 , piece_id=NUZ2IFWCPLA3UF64YDPKP6HH5O4GK6FUYUYY3FVL7ZIVFKJZNUAQ, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:104.36.82.17:4332): protocol: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51046->104.36.82.17:4332: use of closed network connection
Jul 9 20:01:00.622584353 (394h50m35.393225688s) [noise=true , address=104.36.82.34:4546 , piece_id=XROJVC7Q7VCSNA5TSARULFGS753MPQEBDFG6QW2ZX5ENPHKBL3XA, seg_pos={0 0}, upload=315898]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:104.36.82.34:4546): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 20:01:00.622614701 (394h50m35.393256036s) [seg_pos={0 0}, upload=315898]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 20:01:00.859090035 (394h50m35.62973137s) [upload=315898]: Done uploading segments...
Jul 9 20:01:00.859115674 (394h50m35.629757009s) [upload=315898]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.238241123s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X........X.......................X.....................] duration:
5.916964496s
FAIL (2025-07-09 20:00:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.91698721s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
4.887642123s
FAIL (2025-07-09 20:00:09 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.887673923s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........................................X.........X..X...X.] duration:
27.220397921s
FAIL (2025-07-09 19:59:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.220416918s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X........X.......................X....................] duration:
6.109635998s
FAIL (2025-07-09 19:59:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.109659152s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration:
13.138888036s
FAIL (2025-07-09 19:59:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 13.138912432s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........................................X.........X..X...X] duration:
1m0.000993223s
FAIL (2025-07-09 19:59:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001017909s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.830895594s
FAIL (2025-07-09 19:59:04 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.830921572s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................X..X....................] duration:
13.745219956s
FAIL (2025-07-09 19:59:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 13.745238973s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X........X.......................X...................] duration:
5.671285812s
FAIL (2025-07-09 19:58:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.671355516s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration:
59.666517335s
FAIL (2025-07-09 19:58:49 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 59.666543156s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X........X.......................X..................] duration:
5.45462054s
FAIL (2025-07-09 19:58:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.454838147s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
4.909135334s
FAIL (2025-07-09 19:58:16 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.909160912s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X.........X..X...] duration:
5.507045815s
FAIL (2025-07-09 19:58:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.507091332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................X..X..................] duration:
38.964916725s
FAIL (2025-07-09 19:57:41 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 38.96494525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................................X.........X..X..] duration:
24.635636001s
FAIL (2025-07-09 19:57:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.635649376s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X........X.......................X.................] duration:
5.916388191s
FAIL (2025-07-09 19:57:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.916410515s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X.........X..X.] duration:
41.957591697s
FAIL (2025-07-09 19:57:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.95761877s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
4.952862674s
FAIL (2025-07-09 19:57:02 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.952887812s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X........X.......................X................] duration:
6.447963095s
FAIL (2025-07-09 19:56:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.447997321s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
4.735870556s
FAIL (2025-07-09 19:56:39 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.735903519s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X........X.......................X...............] duration:
6.407130574s
FAIL (2025-07-09 19:56:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.407159599s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X..X.................] duration:
23.934602449s
FAIL (2025-07-09 19:56:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 23.934655249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................................X.........X..X] duration:
1m0.000849304s
FAIL (2025-07-09 19:56:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000861127s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
4.70733108s
FAIL (2025-07-09 19:56:04 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.707359605s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................X..X................] duration:
31.884238106s
FAIL (2025-07-09 19:55:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 31.884269295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X........X.......................X..............] duration:
5.897682805s
FAIL (2025-07-09 19:55:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.897705448s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.........X..] duration:
47.436972591s
FAIL (2025-07-09 19:55:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 47.436998923s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.118421661s
FAIL (2025-07-09 19:55:07 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 5.118453513s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X........X.......................X.............] duration:
6.53802575s
FAIL (2025-07-09 19:54:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.538047022s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
4.924847706s
FAIL (2025-07-09 19:54:23 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.924869016s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X........X.......................X............] duration:
6.039155422s
FAIL (2025-07-09 19:54:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.039179628s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration:
26.859719406s
FAIL (2025-07-09 19:54:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 26.85974218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.........X.] duration:
28.241298032s
FAIL (2025-07-09 19:54:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.241445084s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X........X.......................X...........] duration:
6.452517068s
FAIL (2025-07-09 19:53:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.452539681s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.020637296s
FAIL (2025-07-09 19:53:57 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 5.020666874s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
4.714712175s
FAIL (2025-07-09 19:53:34 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.714738596s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.........X] duration:
1m0.004458726s
FAIL (2025-07-09 19:53:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.004470129s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration:
8.726866157s
FAIL (2025-07-09 19:53:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 8.72688834s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X........X.......................X..........] duration:
6.217995723s
FAIL (2025-07-09 19:52:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.218019288s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
4.686219916s
FAIL (2025-07-09 19:52:52 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.686247378s
core
uplink
eu1
|