Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X...X..............X.X.....................................] duration:
21.910744168s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 14:01:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 21.910759427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [........................................X.X..X.XX....X.....X] duration:
1m0.000627095s trace:
https://jaeger.dev.storj.io/trace/52cba4e303e6b349
FAIL (2025-12-30 14:01:56 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000640911s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X....X..] duration:
5.693923162s trace:
https://jaeger.dev.storj.io/trace/4d817cfb8aadba80
FAIL (2025-12-30 14:01:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA) context deadline exceeded
======LOGS======
Dec 30 14:00:02.772968522 (598h31m2.644051567s) [upload=524614]: Starting upload
Dec 30 14:00:02.773139297 (598h31m2.644222362s) [upload=524614]: Done starting upload
Dec 30 14:00:02.773353234 (598h31m2.644436279s) [upload=524614]: Uploading segments...
Dec 30 14:00:02.773558806 (598h31m2.644641851s) [upload=524614]: Got next segment. Inline: false
Dec 30 14:00:02.773634951 (598h31m2.644717996s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:00:02.773643097 (598h31m2.644726142s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:00:03.019958907 (598h31m2.891041952s) [seg_pos={0 0}, upload=524614]: Begin upload segment...
Dec 30 14:00:03.020037657 (598h31m2.891120702s) [seg_pos={0 0}, upload=524614]: Done begin upload segment.
Dec 30 14:00:03.020112439 (598h31m2.891195494s) [noise=true , address=145.79.227.12:4767 , piece_id=YOPBRVWCIZ7BKGFTVBJBWKIVJKXDG2IG7UI3L3FQHK57XH6ONKMA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.022854771 (598h31m2.893937826s) [noise=true , address=145.79.227.47:4266 , piece_id=C2SLTG5XGQFLS6HPXZ2Z6O7RWTHTCLINL3MV5DJ6OIRWYDOXFQVA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.022922149 (598h31m2.894005194s) [noise=true , address=145.79.227.6:4257 , piece_id=B327QDYVQXNSXH4BDTCBXLW6NYXFEVG6N5ABR6JKF7UOLZCUSVSA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.023490652 (598h31m2.894573697s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.023895423 (598h31m2.894978468s) [noise=true , address=77.90.31.14:4662 , piece_id=QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024335862 (598h31m2.895418907s) [noise=true , address=145.79.227.24:4582 , piece_id=FE3UEHILOLSE3FA342F5YXQG54UDFFQDSJA2AJ7TLVVPAIODHR6A, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024669316 (598h31m2.895752361s) [noise=true , address=145.79.227.13:4804 , piece_id=PQEWCFTEHDHNJVPNGPHKHDFO7XF5RCJZUBJL7HFYM2T3RKS5PGDQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024705204 (598h31m2.895788249s) [noise=true , address=145.79.227.19:4078 , piece_id=DD6RVXB5CG5XTTEWJ3BYOY5SGWB47XRCAQJV3WWYG5724Q6R4XEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025152737 (598h31m2.896235782s) [noise=true , address=145.79.227.37:4566 , piece_id=5HI5EQL43NEBHIDQTA5AIXFMQDEBYBTOCZFCRGF6PPL3NYNUKHTA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025466184 (598h31m2.896549239s) [noise=true , address=145.79.227.46:4601 , piece_id=V7OQ54OFJAMBVSZWZECM6BOLYWQSVS4NY2G3PGIU22ZLJVHWZSEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025618444 (598h31m2.896701489s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.026268702 (598h31m2.897351747s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.028015868 (598h31m2.899098913s) [noise=true , address=45.91.116.14:4192 , piece_id=UZGOUJYO34DUJT4QNXWU6CSOGOHBCN747O52VMWNEBQ4SRNM3A7A, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.028702476 (598h31m2.899785521s) [noise=true , address=77.90.31.11:4553 , piece_id=XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.029421776 (598h31m2.900504821s) [noise=true , address=145.79.227.2:4115 , piece_id=24NSEIWHI3EQDQ7SSEMX5YOKV6LH7TGAI3SBHT7MEPOSLLRPZWVQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.030015727 (598h31m2.901098772s) [noise=true , address=145.79.227.36:4517 , piece_id=RA5K645FB4SSNDVX4QPOLI57SZAPIIALIJ47GAKRAZ7MQN5MDJFA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:04.646091554 (598h31m4.517174599s) [upload=524614]: Next returned nil segment
Dec 30 14:00:04.646110130 (598h31m4.517193175s) [upload=524614]: Waiting for error group managing segments...
Dec 30 14:00:05.057348668 (598h31m4.928431713s) [noise=true , address=145.79.227.19:4078 , piece_id=DD6RVXB5CG5XTTEWJ3BYOY5SGWB47XRCAQJV3WWYG5724Q6R4XEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.058881507 (598h31m4.929964542s) [noise=true , address=145.79.227.6:4257 , piece_id=B327QDYVQXNSXH4BDTCBXLW6NYXFEVG6N5ABR6JKF7UOLZCUSVSA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.059987532 (598h31m4.931070577s) [noise=true , address=145.79.227.2:4115 , piece_id=24NSEIWHI3EQDQ7SSEMX5YOKV6LH7TGAI3SBHT7MEPOSLLRPZWVQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.075686234 (598h31m4.946769279s) [noise=true , address=145.79.227.36:4517 , piece_id=RA5K645FB4SSNDVX4QPOLI57SZAPIIALIJ47GAKRAZ7MQN5MDJFA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.100463274 (598h31m4.971546319s) [noise=true , address=145.79.227.12:4767 , piece_id=YOPBRVWCIZ7BKGFTVBJBWKIVJKXDG2IG7UI3L3FQHK57XH6ONKMA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.124259596 (598h31m4.995342651s) [noise=true , address=145.79.227.37:4566 , piece_id=5HI5EQL43NEBHIDQTA5AIXFMQDEBYBTOCZFCRGF6PPL3NYNUKHTA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.158003218 (598h31m5.029086264s) [noise=true , address=145.79.227.47:4266 , piece_id=C2SLTG5XGQFLS6HPXZ2Z6O7RWTHTCLINL3MV5DJ6OIRWYDOXFQVA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.161485028 (598h31m5.032568073s) [noise=true , address=145.79.227.13:4804 , piece_id=PQEWCFTEHDHNJVPNGPHKHDFO7XF5RCJZUBJL7HFYM2T3RKS5PGDQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.163717058 (598h31m5.034800104s) [noise=true , address=145.79.227.24:4582 , piece_id=FE3UEHILOLSE3FA342F5YXQG54UDFFQDSJA2AJ7TLVVPAIODHR6A, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.171656416 (598h31m5.042739461s) [noise=true , address=145.79.227.46:4601 , piece_id=V7OQ54OFJAMBVSZWZECM6BOLYWQSVS4NY2G3PGIU22ZLJVHWZSEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.298521466 (598h31m5.169604511s) [noise=true , address=77.90.31.11:4553 , piece_id=XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.385624783 (598h31m5.256707828s) [noise=true , address=77.90.31.14:4662 , piece_id=QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.821554266 (598h31m5.692637321s) [noise=true , address=45.91.116.14:4192 , piece_id=UZGOUJYO34DUJT4QNXWU6CSOGOHBCN747O52VMWNEBQ4SRNM3A7A, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.821580616 (598h31m5.692663661s) [seg_pos={0 0}, upload=524614]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:00:05.821941493 (598h31m5.693024538s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 30 14:00:05.821953055 (598h31m5.693036101s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.822413342 (598h31m5.693496377s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection
Dec 30 14:00:05.822637298 (598h31m5.693720403s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.823256277 (598h31m5.694339332s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 14:00:05.823268090 (598h31m5.694351145s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.823294950 (598h31m5.694377985s) [seg_pos={0 0}, upload=524614]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:00:05.823400852 (598h31m5.694483897s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:00:05.823408727 (598h31m5.694491783s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:00:06.125499242 (598h31m5.996582287s) [upload=524614]: Done uploading segments...
Dec 30 14:00:06.125522877 (598h31m5.996605922s) [upload=524614]: Upload finished. err: <nil>
Duration: 5.693963629s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.XX.XXX] duration:
5.001765861s trace:
https://jaeger.dev.storj.io/trace/313ce8b50cecdc72
FAIL (2025-12-30 14:01:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001788305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X...X..............X.X....................................] duration:
41.400223203s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 14:01:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 41.400250195s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.XX.XX] duration:
5.001213962s trace:
https://jaeger.dev.storj.io/trace/713da798f1771e42
FAIL (2025-12-30 14:01:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001237716s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [XX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..XX....X] duration:
30.001664264s trace:
https://jaeger.dev.storj.io/trace/74ba0dead003e2ac
FAIL (2025-12-30 14:01:14 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001676527s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
10.853819198s trace:
https://jaeger.dev.storj.io/trace/7aa21b18ea57a13d
FAIL (2025-12-30 14:01:09 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.853847433s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [......X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X....X.] duration:
5.856384741s trace:
https://jaeger.dev.storj.io/trace/4d817cfb8aadba80
FAIL (2025-12-30 14:00:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA) context deadline exceeded
======LOGS======
Dec 30 14:00:02.772968522 (598h31m2.644051567s) [upload=524614]: Starting upload
Dec 30 14:00:02.773139297 (598h31m2.644222362s) [upload=524614]: Done starting upload
Dec 30 14:00:02.773353234 (598h31m2.644436279s) [upload=524614]: Uploading segments...
Dec 30 14:00:02.773558806 (598h31m2.644641851s) [upload=524614]: Got next segment. Inline: false
Dec 30 14:00:02.773634951 (598h31m2.644717996s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:00:02.773643097 (598h31m2.644726142s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:00:03.019958907 (598h31m2.891041952s) [seg_pos={0 0}, upload=524614]: Begin upload segment...
Dec 30 14:00:03.020037657 (598h31m2.891120702s) [seg_pos={0 0}, upload=524614]: Done begin upload segment.
Dec 30 14:00:03.020112439 (598h31m2.891195494s) [noise=true , address=145.79.227.12:4767 , piece_id=YOPBRVWCIZ7BKGFTVBJBWKIVJKXDG2IG7UI3L3FQHK57XH6ONKMA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.022854771 (598h31m2.893937826s) [noise=true , address=145.79.227.47:4266 , piece_id=C2SLTG5XGQFLS6HPXZ2Z6O7RWTHTCLINL3MV5DJ6OIRWYDOXFQVA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.022922149 (598h31m2.894005194s) [noise=true , address=145.79.227.6:4257 , piece_id=B327QDYVQXNSXH4BDTCBXLW6NYXFEVG6N5ABR6JKF7UOLZCUSVSA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.023490652 (598h31m2.894573697s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.023895423 (598h31m2.894978468s) [noise=true , address=77.90.31.14:4662 , piece_id=QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024335862 (598h31m2.895418907s) [noise=true , address=145.79.227.24:4582 , piece_id=FE3UEHILOLSE3FA342F5YXQG54UDFFQDSJA2AJ7TLVVPAIODHR6A, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024669316 (598h31m2.895752361s) [noise=true , address=145.79.227.13:4804 , piece_id=PQEWCFTEHDHNJVPNGPHKHDFO7XF5RCJZUBJL7HFYM2T3RKS5PGDQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024705204 (598h31m2.895788249s) [noise=true , address=145.79.227.19:4078 , piece_id=DD6RVXB5CG5XTTEWJ3BYOY5SGWB47XRCAQJV3WWYG5724Q6R4XEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025152737 (598h31m2.896235782s) [noise=true , address=145.79.227.37:4566 , piece_id=5HI5EQL43NEBHIDQTA5AIXFMQDEBYBTOCZFCRGF6PPL3NYNUKHTA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025466184 (598h31m2.896549239s) [noise=true , address=145.79.227.46:4601 , piece_id=V7OQ54OFJAMBVSZWZECM6BOLYWQSVS4NY2G3PGIU22ZLJVHWZSEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025618444 (598h31m2.896701489s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.026268702 (598h31m2.897351747s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.028015868 (598h31m2.899098913s) [noise=true , address=45.91.116.14:4192 , piece_id=UZGOUJYO34DUJT4QNXWU6CSOGOHBCN747O52VMWNEBQ4SRNM3A7A, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.028702476 (598h31m2.899785521s) [noise=true , address=77.90.31.11:4553 , piece_id=XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.029421776 (598h31m2.900504821s) [noise=true , address=145.79.227.2:4115 , piece_id=24NSEIWHI3EQDQ7SSEMX5YOKV6LH7TGAI3SBHT7MEPOSLLRPZWVQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.030015727 (598h31m2.901098772s) [noise=true , address=145.79.227.36:4517 , piece_id=RA5K645FB4SSNDVX4QPOLI57SZAPIIALIJ47GAKRAZ7MQN5MDJFA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:04.646091554 (598h31m4.517174599s) [upload=524614]: Next returned nil segment
Dec 30 14:00:04.646110130 (598h31m4.517193175s) [upload=524614]: Waiting for error group managing segments...
Dec 30 14:00:05.057348668 (598h31m4.928431713s) [noise=true , address=145.79.227.19:4078 , piece_id=DD6RVXB5CG5XTTEWJ3BYOY5SGWB47XRCAQJV3WWYG5724Q6R4XEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.058881507 (598h31m4.929964542s) [noise=true , address=145.79.227.6:4257 , piece_id=B327QDYVQXNSXH4BDTCBXLW6NYXFEVG6N5ABR6JKF7UOLZCUSVSA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.059987532 (598h31m4.931070577s) [noise=true , address=145.79.227.2:4115 , piece_id=24NSEIWHI3EQDQ7SSEMX5YOKV6LH7TGAI3SBHT7MEPOSLLRPZWVQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.075686234 (598h31m4.946769279s) [noise=true , address=145.79.227.36:4517 , piece_id=RA5K645FB4SSNDVX4QPOLI57SZAPIIALIJ47GAKRAZ7MQN5MDJFA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.100463274 (598h31m4.971546319s) [noise=true , address=145.79.227.12:4767 , piece_id=YOPBRVWCIZ7BKGFTVBJBWKIVJKXDG2IG7UI3L3FQHK57XH6ONKMA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.124259596 (598h31m4.995342651s) [noise=true , address=145.79.227.37:4566 , piece_id=5HI5EQL43NEBHIDQTA5AIXFMQDEBYBTOCZFCRGF6PPL3NYNUKHTA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.158003218 (598h31m5.029086264s) [noise=true , address=145.79.227.47:4266 , piece_id=C2SLTG5XGQFLS6HPXZ2Z6O7RWTHTCLINL3MV5DJ6OIRWYDOXFQVA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.161485028 (598h31m5.032568073s) [noise=true , address=145.79.227.13:4804 , piece_id=PQEWCFTEHDHNJVPNGPHKHDFO7XF5RCJZUBJL7HFYM2T3RKS5PGDQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.163717058 (598h31m5.034800104s) [noise=true , address=145.79.227.24:4582 , piece_id=FE3UEHILOLSE3FA342F5YXQG54UDFFQDSJA2AJ7TLVVPAIODHR6A, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.171656416 (598h31m5.042739461s) [noise=true , address=145.79.227.46:4601 , piece_id=V7OQ54OFJAMBVSZWZECM6BOLYWQSVS4NY2G3PGIU22ZLJVHWZSEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.298521466 (598h31m5.169604511s) [noise=true , address=77.90.31.11:4553 , piece_id=XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.385624783 (598h31m5.256707828s) [noise=true , address=77.90.31.14:4662 , piece_id=QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.821554266 (598h31m5.692637321s) [noise=true , address=45.91.116.14:4192 , piece_id=UZGOUJYO34DUJT4QNXWU6CSOGOHBCN747O52VMWNEBQ4SRNM3A7A, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.821580616 (598h31m5.692663661s) [seg_pos={0 0}, upload=524614]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:00:05.821941493 (598h31m5.693024538s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 30 14:00:05.821953055 (598h31m5.693036101s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.822413342 (598h31m5.693496377s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection
Dec 30 14:00:05.822637298 (598h31m5.693720403s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.823256277 (598h31m5.694339332s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 14:00:05.823268090 (598h31m5.694351145s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.823294950 (598h31m5.694377985s) [seg_pos={0 0}, upload=524614]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:00:05.823400852 (598h31m5.694483897s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:00:05.823408727 (598h31m5.694491783s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:00:06.125499242 (598h31m5.996582287s) [upload=524614]: Done uploading segments...
Dec 30 14:00:06.125522877 (598h31m5.996605922s) [upload=524614]: Upload finished. err: <nil>
Duration: 5.856414858s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........................................X.X..X.XX....X.....] duration:
5.944351333s trace:
https://jaeger.dev.storj.io/trace/110e2ffb085aa52e
FAIL (2025-12-30 14:00:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.944366002s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........................................X.X..X.XX....X....] duration:
52.540248217s trace:
https://jaeger.dev.storj.io/trace/110e2ffb085aa52e
FAIL (2025-12-30 14:00:41 UTC)
orbiter warning: context deadline exceeded
Duration: 52.540275668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.......X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X....X] duration:
30.003346548s trace:
https://jaeger.dev.storj.io/trace/4d817cfb8aadba80
FAIL (2025-12-30 14:00:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA) context deadline exceeded
======LOGS======
Dec 30 14:00:02.772968522 (598h31m2.644051567s) [upload=524614]: Starting upload
Dec 30 14:00:02.773139297 (598h31m2.644222362s) [upload=524614]: Done starting upload
Dec 30 14:00:02.773353234 (598h31m2.644436279s) [upload=524614]: Uploading segments...
Dec 30 14:00:02.773558806 (598h31m2.644641851s) [upload=524614]: Got next segment. Inline: false
Dec 30 14:00:02.773634951 (598h31m2.644717996s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:00:02.773643097 (598h31m2.644726142s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:00:03.019958907 (598h31m2.891041952s) [seg_pos={0 0}, upload=524614]: Begin upload segment...
Dec 30 14:00:03.020037657 (598h31m2.891120702s) [seg_pos={0 0}, upload=524614]: Done begin upload segment.
Dec 30 14:00:03.020112439 (598h31m2.891195494s) [noise=true , address=145.79.227.12:4767 , piece_id=YOPBRVWCIZ7BKGFTVBJBWKIVJKXDG2IG7UI3L3FQHK57XH6ONKMA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.022854771 (598h31m2.893937826s) [noise=true , address=145.79.227.47:4266 , piece_id=C2SLTG5XGQFLS6HPXZ2Z6O7RWTHTCLINL3MV5DJ6OIRWYDOXFQVA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.022922149 (598h31m2.894005194s) [noise=true , address=145.79.227.6:4257 , piece_id=B327QDYVQXNSXH4BDTCBXLW6NYXFEVG6N5ABR6JKF7UOLZCUSVSA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.023490652 (598h31m2.894573697s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.023895423 (598h31m2.894978468s) [noise=true , address=77.90.31.14:4662 , piece_id=QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024335862 (598h31m2.895418907s) [noise=true , address=145.79.227.24:4582 , piece_id=FE3UEHILOLSE3FA342F5YXQG54UDFFQDSJA2AJ7TLVVPAIODHR6A, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024669316 (598h31m2.895752361s) [noise=true , address=145.79.227.13:4804 , piece_id=PQEWCFTEHDHNJVPNGPHKHDFO7XF5RCJZUBJL7HFYM2T3RKS5PGDQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.024705204 (598h31m2.895788249s) [noise=true , address=145.79.227.19:4078 , piece_id=DD6RVXB5CG5XTTEWJ3BYOY5SGWB47XRCAQJV3WWYG5724Q6R4XEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025152737 (598h31m2.896235782s) [noise=true , address=145.79.227.37:4566 , piece_id=5HI5EQL43NEBHIDQTA5AIXFMQDEBYBTOCZFCRGF6PPL3NYNUKHTA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025466184 (598h31m2.896549239s) [noise=true , address=145.79.227.46:4601 , piece_id=V7OQ54OFJAMBVSZWZECM6BOLYWQSVS4NY2G3PGIU22ZLJVHWZSEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.025618444 (598h31m2.896701489s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.026268702 (598h31m2.897351747s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.028015868 (598h31m2.899098913s) [noise=true , address=45.91.116.14:4192 , piece_id=UZGOUJYO34DUJT4QNXWU6CSOGOHBCN747O52VMWNEBQ4SRNM3A7A, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.028702476 (598h31m2.899785521s) [noise=true , address=77.90.31.11:4553 , piece_id=XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.029421776 (598h31m2.900504821s) [noise=true , address=145.79.227.2:4115 , piece_id=24NSEIWHI3EQDQ7SSEMX5YOKV6LH7TGAI3SBHT7MEPOSLLRPZWVQ, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:03.030015727 (598h31m2.901098772s) [noise=true , address=145.79.227.36:4517 , piece_id=RA5K645FB4SSNDVX4QPOLI57SZAPIIALIJ47GAKRAZ7MQN5MDJFA, seg_pos={0 0}, upload=524614]: Uploading piece...
Dec 30 14:00:04.646091554 (598h31m4.517174599s) [upload=524614]: Next returned nil segment
Dec 30 14:00:04.646110130 (598h31m4.517193175s) [upload=524614]: Waiting for error group managing segments...
Dec 30 14:00:05.057348668 (598h31m4.928431713s) [noise=true , address=145.79.227.19:4078 , piece_id=DD6RVXB5CG5XTTEWJ3BYOY5SGWB47XRCAQJV3WWYG5724Q6R4XEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.058881507 (598h31m4.929964542s) [noise=true , address=145.79.227.6:4257 , piece_id=B327QDYVQXNSXH4BDTCBXLW6NYXFEVG6N5ABR6JKF7UOLZCUSVSA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.059987532 (598h31m4.931070577s) [noise=true , address=145.79.227.2:4115 , piece_id=24NSEIWHI3EQDQ7SSEMX5YOKV6LH7TGAI3SBHT7MEPOSLLRPZWVQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.075686234 (598h31m4.946769279s) [noise=true , address=145.79.227.36:4517 , piece_id=RA5K645FB4SSNDVX4QPOLI57SZAPIIALIJ47GAKRAZ7MQN5MDJFA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.100463274 (598h31m4.971546319s) [noise=true , address=145.79.227.12:4767 , piece_id=YOPBRVWCIZ7BKGFTVBJBWKIVJKXDG2IG7UI3L3FQHK57XH6ONKMA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.124259596 (598h31m4.995342651s) [noise=true , address=145.79.227.37:4566 , piece_id=5HI5EQL43NEBHIDQTA5AIXFMQDEBYBTOCZFCRGF6PPL3NYNUKHTA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.158003218 (598h31m5.029086264s) [noise=true , address=145.79.227.47:4266 , piece_id=C2SLTG5XGQFLS6HPXZ2Z6O7RWTHTCLINL3MV5DJ6OIRWYDOXFQVA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.161485028 (598h31m5.032568073s) [noise=true , address=145.79.227.13:4804 , piece_id=PQEWCFTEHDHNJVPNGPHKHDFO7XF5RCJZUBJL7HFYM2T3RKS5PGDQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.163717058 (598h31m5.034800104s) [noise=true , address=145.79.227.24:4582 , piece_id=FE3UEHILOLSE3FA342F5YXQG54UDFFQDSJA2AJ7TLVVPAIODHR6A, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.171656416 (598h31m5.042739461s) [noise=true , address=145.79.227.46:4601 , piece_id=V7OQ54OFJAMBVSZWZECM6BOLYWQSVS4NY2G3PGIU22ZLJVHWZSEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.298521466 (598h31m5.169604511s) [noise=true , address=77.90.31.11:4553 , piece_id=XAFTRILI3UCX7SYH27TSJGII6PFVVN737RJBENNX23DFMSKAQTEA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.385624783 (598h31m5.256707828s) [noise=true , address=77.90.31.14:4662 , piece_id=QKEOGEVSSIL2GVJ77PZ44D6IZB4V54JBIE6ARYNE2NOCSITDVZVA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.821554266 (598h31m5.692637321s) [noise=true , address=45.91.116.14:4192 , piece_id=UZGOUJYO34DUJT4QNXWU6CSOGOHBCN747O52VMWNEBQ4SRNM3A7A, seg_pos={0 0}, upload=524614]: Done uploading piece. err: <nil>
Dec 30 14:00:05.821580616 (598h31m5.692663661s) [seg_pos={0 0}, upload=524614]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:00:05.821941493 (598h31m5.693024538s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 30 14:00:05.821953055 (598h31m5.693036101s) [noise=true , address=45.91.116.8:4046 , piece_id=3GB66LSLMRO5RS4H5QXBAMF32NMCME3NWE4E2DIRTU5KQ5HTPXDQ, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.822413342 (598h31m5.693496377s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53338->45.91.116.6:4974: use of closed network connection
Dec 30 14:00:05.822637298 (598h31m5.693720403s) [noise=true , address=45.91.116.6:4974 , piece_id=JZ4NZLM7IQSRCGSM34EJAI6Z3BZQLUFBPQDVNWE3DDRZUG2BJS5Q, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.823256277 (598h31m5.694339332s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 14:00:05.823268090 (598h31m5.694351145s) [noise=true , address=145.79.227.34:4906 , piece_id=3NFKWO2R7JKLL7GMHQAEATBMKBCPDJBI66BQWC2CHSFZDVPQNKKA, seg_pos={0 0}, upload=524614]: Long tail cancellation due to optimal threshold
Dec 30 14:00:05.823294950 (598h31m5.694377985s) [seg_pos={0 0}, upload=524614]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:00:05.823400852 (598h31m5.694483897s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:00:05.823408727 (598h31m5.694491783s) [upload=524614]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:00:06.125499242 (598h31m5.996582287s) [upload=524614]: Done uploading segments...
Dec 30 14:00:06.125522877 (598h31m5.996605922s) [upload=524614]: Upload finished. err: <nil>
Duration: 30.003370733s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X...X..............X.X...................................] duration:
40.42794353s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 14:00:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 40.427965522s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..XX....] duration:
22.797053637s trace:
https://jaeger.dev.storj.io/trace/58698a2027ff0c3c
FAIL (2025-12-30 14:00:19 UTC)
orbiter warning: context deadline exceeded
Duration: 22.797436206s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.XX.X] duration:
5.001273443s trace:
https://jaeger.dev.storj.io/trace/3bd91a50e16b41d0
FAIL (2025-12-30 14:00:10 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001331375s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...........................................X.X..X.XX....X...] duration:
16.17170671s trace:
https://jaeger.dev.storj.io/trace/110e2ffb085aa52e
FAIL (2025-12-30 13:59:44 UTC)
orbiter warning: context deadline exceeded
Duration: 16.171734704s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X...X..............X.X..................................] duration:
40.699617776s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:59:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 40.699634928s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..XX...] duration:
22.637128011s trace:
https://jaeger.dev.storj.io/trace/58698a2027ff0c3c
FAIL (2025-12-30 13:59:27 UTC)
orbiter warning: context deadline exceeded
Duration: 22.637168268s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [........X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X....] duration:
5.938427317s trace:
https://jaeger.dev.storj.io/trace/64e00f8bbf96832e
FAIL (2025-12-30 13:59:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q) context deadline exceeded
======LOGS======
Dec 30 13:55:29.527888000 (598h26m29.398971035s) [upload=524552]: Starting upload
Dec 30 13:55:29.527942653 (598h26m29.399025688s) [upload=524552]: Done starting upload
Dec 30 13:55:29.528082490 (598h26m29.399165535s) [upload=524552]: Uploading segments...
Dec 30 13:55:29.528210924 (598h26m29.39929397s) [upload=524552]: Got next segment. Inline: false
Dec 30 13:55:29.528249157 (598h26m29.399332202s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:55:29.528255750 (598h26m29.399338785s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:55:29.775747671 (598h26m29.646830706s) [seg_pos={0 0}, upload=524552]: Begin upload segment...
Dec 30 13:55:29.775791585 (598h26m29.64687462s) [seg_pos={0 0}, upload=524552]: Done begin upload segment.
Dec 30 13:55:29.775816613 (598h26m29.646899648s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.775871357 (598h26m29.646954392s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.776666852 (598h26m29.647749887s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777101459 (598h26m29.648184494s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777497202 (598h26m29.648580237s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777927573 (598h26m29.649010618s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778175765 (598h26m29.64925882s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778474233 (598h26m29.649557278s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778595244 (598h26m29.649678279s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778932897 (598h26m29.650015942s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779032897 (598h26m29.650115943s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779350602 (598h26m29.650433647s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779797082 (598h26m29.650880117s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780239865 (598h26m29.65132291s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780740239 (598h26m29.651823284s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.781264657 (598h26m29.652347692s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:31.494971108 (598h26m31.366054143s) [upload=524552]: Next returned nil segment
Dec 30 13:55:31.494995605 (598h26m31.36607864s) [upload=524552]: Waiting for error group managing segments...
Dec 30 13:55:31.788446429 (598h26m31.659529474s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.791836153 (598h26m31.662919188s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.798297788 (598h26m31.669380833s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.801691269 (598h26m31.672774304s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851643325 (598h26m31.72272636s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851766319 (598h26m31.722849364s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.861568735 (598h26m31.73265177s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.116844929 (598h26m31.987927964s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.187932641 (598h26m32.059015676s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.209901162 (598h26m32.080984208s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.224473169 (598h26m32.095556205s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.231189418 (598h26m32.102272453s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270421372 (598h26m32.141504417s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270444706 (598h26m32.141527741s) [seg_pos={0 0}, upload=524552]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:55:32.270661479 (598h26m32.141744524s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270672410 (598h26m32.141755455s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270709160 (598h26m32.141792205s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270717525 (598h26m32.14180056s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270997779 (598h26m32.142080844s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:52186->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 13:55:32.271008560 (598h26m32.142091625s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.271030461 (598h26m32.142113506s) [seg_pos={0 0}, upload=524552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:55:32.271116966 (598h26m32.142200011s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:55:32.271124250 (598h26m32.142207295s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:55:32.571519748 (598h26m32.442602793s) [upload=524552]: Done uploading segments...
Dec 30 13:55:32.571542702 (598h26m32.442625737s) [upload=524552]: Upload finished. err: <nil>
Duration: 5.938455711s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.XX.] duration:
3.96679372s trace:
https://jaeger.dev.storj.io/trace/7e1ddd6ee91ba569
FAIL (2025-12-30 13:59:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.966822234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X...X..............X.X.................................] duration:
36.494229414s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:58:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 36.49425884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..XX..] duration:
24.217391045s trace:
https://jaeger.dev.storj.io/trace/58698a2027ff0c3c
FAIL (2025-12-30 13:58:51 UTC)
orbiter warning: context deadline exceeded
Duration: 24.217417286s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [............................................X.X..X.XX....X..] duration:
16.380546151s trace:
https://jaeger.dev.storj.io/trace/110e2ffb085aa52e
FAIL (2025-12-30 13:58:44 UTC)
orbiter warning: context deadline exceeded
Duration: 16.380569036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..XX.] duration:
20.284549739s trace:
https://jaeger.dev.storj.io/trace/58698a2027ff0c3c
FAIL (2025-12-30 13:58:10 UTC)
orbiter warning: context deadline exceeded
Duration: 20.284578413s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X...X..............X.X................................] duration:
44.650236425s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:57:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 44.650264126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.XX] duration:
5.001518877s trace:
https://jaeger.dev.storj.io/trace/7e1ddd6ee91ba569
FAIL (2025-12-30 13:57:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00154116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.........X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X...] duration:
5.953527172s trace:
https://jaeger.dev.storj.io/trace/64e00f8bbf96832e
FAIL (2025-12-30 13:57:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q) context deadline exceeded
======LOGS======
Dec 30 13:55:29.527888000 (598h26m29.398971035s) [upload=524552]: Starting upload
Dec 30 13:55:29.527942653 (598h26m29.399025688s) [upload=524552]: Done starting upload
Dec 30 13:55:29.528082490 (598h26m29.399165535s) [upload=524552]: Uploading segments...
Dec 30 13:55:29.528210924 (598h26m29.39929397s) [upload=524552]: Got next segment. Inline: false
Dec 30 13:55:29.528249157 (598h26m29.399332202s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:55:29.528255750 (598h26m29.399338785s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:55:29.775747671 (598h26m29.646830706s) [seg_pos={0 0}, upload=524552]: Begin upload segment...
Dec 30 13:55:29.775791585 (598h26m29.64687462s) [seg_pos={0 0}, upload=524552]: Done begin upload segment.
Dec 30 13:55:29.775816613 (598h26m29.646899648s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.775871357 (598h26m29.646954392s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.776666852 (598h26m29.647749887s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777101459 (598h26m29.648184494s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777497202 (598h26m29.648580237s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777927573 (598h26m29.649010618s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778175765 (598h26m29.64925882s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778474233 (598h26m29.649557278s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778595244 (598h26m29.649678279s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778932897 (598h26m29.650015942s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779032897 (598h26m29.650115943s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779350602 (598h26m29.650433647s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779797082 (598h26m29.650880117s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780239865 (598h26m29.65132291s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780740239 (598h26m29.651823284s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.781264657 (598h26m29.652347692s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:31.494971108 (598h26m31.366054143s) [upload=524552]: Next returned nil segment
Dec 30 13:55:31.494995605 (598h26m31.36607864s) [upload=524552]: Waiting for error group managing segments...
Dec 30 13:55:31.788446429 (598h26m31.659529474s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.791836153 (598h26m31.662919188s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.798297788 (598h26m31.669380833s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.801691269 (598h26m31.672774304s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851643325 (598h26m31.72272636s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851766319 (598h26m31.722849364s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.861568735 (598h26m31.73265177s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.116844929 (598h26m31.987927964s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.187932641 (598h26m32.059015676s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.209901162 (598h26m32.080984208s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.224473169 (598h26m32.095556205s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.231189418 (598h26m32.102272453s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270421372 (598h26m32.141504417s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270444706 (598h26m32.141527741s) [seg_pos={0 0}, upload=524552]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:55:32.270661479 (598h26m32.141744524s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270672410 (598h26m32.141755455s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270709160 (598h26m32.141792205s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270717525 (598h26m32.14180056s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270997779 (598h26m32.142080844s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:52186->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 13:55:32.271008560 (598h26m32.142091625s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.271030461 (598h26m32.142113506s) [seg_pos={0 0}, upload=524552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:55:32.271116966 (598h26m32.142200011s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:55:32.271124250 (598h26m32.142207295s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:55:32.571519748 (598h26m32.442602793s) [upload=524552]: Done uploading segments...
Dec 30 13:55:32.571542702 (598h26m32.442625737s) [upload=524552]: Upload finished. err: <nil>
Duration: 5.953547652s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.............................................X.X..X.XX....X.] duration:
19.354140469s trace:
https://jaeger.dev.storj.io/trace/110e2ffb085aa52e
FAIL (2025-12-30 13:57:51 UTC)
orbiter warning: context deadline exceeded
Duration: 19.354183602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..XX] duration:
30.001379953s trace:
https://jaeger.dev.storj.io/trace/58698a2027ff0c3c
FAIL (2025-12-30 13:57:38 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001433093s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..........X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X..] duration:
5.771898324s trace:
https://jaeger.dev.storj.io/trace/64e00f8bbf96832e
FAIL (2025-12-30 13:57:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q) context deadline exceeded
======LOGS======
Dec 30 13:55:29.527888000 (598h26m29.398971035s) [upload=524552]: Starting upload
Dec 30 13:55:29.527942653 (598h26m29.399025688s) [upload=524552]: Done starting upload
Dec 30 13:55:29.528082490 (598h26m29.399165535s) [upload=524552]: Uploading segments...
Dec 30 13:55:29.528210924 (598h26m29.39929397s) [upload=524552]: Got next segment. Inline: false
Dec 30 13:55:29.528249157 (598h26m29.399332202s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:55:29.528255750 (598h26m29.399338785s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:55:29.775747671 (598h26m29.646830706s) [seg_pos={0 0}, upload=524552]: Begin upload segment...
Dec 30 13:55:29.775791585 (598h26m29.64687462s) [seg_pos={0 0}, upload=524552]: Done begin upload segment.
Dec 30 13:55:29.775816613 (598h26m29.646899648s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.775871357 (598h26m29.646954392s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.776666852 (598h26m29.647749887s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777101459 (598h26m29.648184494s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777497202 (598h26m29.648580237s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777927573 (598h26m29.649010618s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778175765 (598h26m29.64925882s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778474233 (598h26m29.649557278s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778595244 (598h26m29.649678279s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778932897 (598h26m29.650015942s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779032897 (598h26m29.650115943s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779350602 (598h26m29.650433647s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779797082 (598h26m29.650880117s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780239865 (598h26m29.65132291s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780740239 (598h26m29.651823284s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.781264657 (598h26m29.652347692s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:31.494971108 (598h26m31.366054143s) [upload=524552]: Next returned nil segment
Dec 30 13:55:31.494995605 (598h26m31.36607864s) [upload=524552]: Waiting for error group managing segments...
Dec 30 13:55:31.788446429 (598h26m31.659529474s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.791836153 (598h26m31.662919188s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.798297788 (598h26m31.669380833s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.801691269 (598h26m31.672774304s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851643325 (598h26m31.72272636s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851766319 (598h26m31.722849364s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.861568735 (598h26m31.73265177s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.116844929 (598h26m31.987927964s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.187932641 (598h26m32.059015676s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.209901162 (598h26m32.080984208s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.224473169 (598h26m32.095556205s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.231189418 (598h26m32.102272453s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270421372 (598h26m32.141504417s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270444706 (598h26m32.141527741s) [seg_pos={0 0}, upload=524552]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:55:32.270661479 (598h26m32.141744524s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270672410 (598h26m32.141755455s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270709160 (598h26m32.141792205s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270717525 (598h26m32.14180056s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270997779 (598h26m32.142080844s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:52186->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 13:55:32.271008560 (598h26m32.142091625s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.271030461 (598h26m32.142113506s) [seg_pos={0 0}, upload=524552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:55:32.271116966 (598h26m32.142200011s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:55:32.271124250 (598h26m32.142207295s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:55:32.571519748 (598h26m32.442602793s) [upload=524552]: Done uploading segments...
Dec 30 13:55:32.571542702 (598h26m32.442625737s) [upload=524552]: Upload finished. err: <nil>
Duration: 5.771925676s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..............................................X.X..X.XX....X] duration:
1m0.000771953s trace:
https://jaeger.dev.storj.io/trace/110e2ffb085aa52e
FAIL (2025-12-30 13:57:32 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000794414s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.XXXXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.X] duration:
5.001587472s trace:
https://jaeger.dev.storj.io/trace/46ace93e7dc645ae
FAIL (2025-12-30 13:57:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0015966s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.X...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..X] duration:
30.001413789s trace:
https://jaeger.dev.storj.io/trace/6339f0983d8b272c
FAIL (2025-12-30 13:57:06 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001425761s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...........X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X.] duration:
6.061333839s trace:
https://jaeger.dev.storj.io/trace/64e00f8bbf96832e
FAIL (2025-12-30 13:56:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q) context deadline exceeded
======LOGS======
Dec 30 13:55:29.527888000 (598h26m29.398971035s) [upload=524552]: Starting upload
Dec 30 13:55:29.527942653 (598h26m29.399025688s) [upload=524552]: Done starting upload
Dec 30 13:55:29.528082490 (598h26m29.399165535s) [upload=524552]: Uploading segments...
Dec 30 13:55:29.528210924 (598h26m29.39929397s) [upload=524552]: Got next segment. Inline: false
Dec 30 13:55:29.528249157 (598h26m29.399332202s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:55:29.528255750 (598h26m29.399338785s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:55:29.775747671 (598h26m29.646830706s) [seg_pos={0 0}, upload=524552]: Begin upload segment...
Dec 30 13:55:29.775791585 (598h26m29.64687462s) [seg_pos={0 0}, upload=524552]: Done begin upload segment.
Dec 30 13:55:29.775816613 (598h26m29.646899648s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.775871357 (598h26m29.646954392s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.776666852 (598h26m29.647749887s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777101459 (598h26m29.648184494s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777497202 (598h26m29.648580237s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777927573 (598h26m29.649010618s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778175765 (598h26m29.64925882s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778474233 (598h26m29.649557278s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778595244 (598h26m29.649678279s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778932897 (598h26m29.650015942s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779032897 (598h26m29.650115943s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779350602 (598h26m29.650433647s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779797082 (598h26m29.650880117s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780239865 (598h26m29.65132291s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780740239 (598h26m29.651823284s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.781264657 (598h26m29.652347692s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:31.494971108 (598h26m31.366054143s) [upload=524552]: Next returned nil segment
Dec 30 13:55:31.494995605 (598h26m31.36607864s) [upload=524552]: Waiting for error group managing segments...
Dec 30 13:55:31.788446429 (598h26m31.659529474s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.791836153 (598h26m31.662919188s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.798297788 (598h26m31.669380833s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.801691269 (598h26m31.672774304s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851643325 (598h26m31.72272636s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851766319 (598h26m31.722849364s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.861568735 (598h26m31.73265177s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.116844929 (598h26m31.987927964s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.187932641 (598h26m32.059015676s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.209901162 (598h26m32.080984208s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.224473169 (598h26m32.095556205s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.231189418 (598h26m32.102272453s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270421372 (598h26m32.141504417s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270444706 (598h26m32.141527741s) [seg_pos={0 0}, upload=524552]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:55:32.270661479 (598h26m32.141744524s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270672410 (598h26m32.141755455s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270709160 (598h26m32.141792205s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270717525 (598h26m32.14180056s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270997779 (598h26m32.142080844s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:52186->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 13:55:32.271008560 (598h26m32.142091625s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.271030461 (598h26m32.142113506s) [seg_pos={0 0}, upload=524552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:55:32.271116966 (598h26m32.142200011s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:55:32.271124250 (598h26m32.142207295s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:55:32.571519748 (598h26m32.442602793s) [upload=524552]: Done uploading segments...
Dec 30 13:55:32.571542702 (598h26m32.442625737s) [upload=524552]: Upload finished. err: <nil>
Duration: 6.061365648s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X...X..............X.X...............................] duration:
15.408232773s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:56:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 15.408261759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX.] duration:
3.776695353s trace:
https://jaeger.dev.storj.io/trace/7571848bb69e4490
FAIL (2025-12-30 13:56:14 UTC)
orbiter warning: context deadline exceeded
Duration: 3.776743414s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [..X...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X..] duration:
20.010820684s trace:
https://jaeger.dev.storj.io/trace/6d8ec002a4742b07
FAIL (2025-12-30 13:56:11 UTC)
orbiter warning: context deadline exceeded
Duration: 20.01083973s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............................................X.X..X.XX....] duration:
39.22179257s trace:
https://jaeger.dev.storj.io/trace/5dea0761a983230a
FAIL (2025-12-30 13:56:01 UTC)
orbiter warning: context deadline exceeded
Duration: 39.22181896s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [X.......X...X..............X.X..............................] duration:
55.935455864s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:56:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 55.935485953s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [............X.XX....XX..XXX.......XXXX.XX.XXX...X...X......X] duration:
30.002391696s trace:
https://jaeger.dev.storj.io/trace/64e00f8bbf96832e
FAIL (2025-12-30 13:55:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q) context deadline exceeded
======LOGS======
Dec 30 13:55:29.527888000 (598h26m29.398971035s) [upload=524552]: Starting upload
Dec 30 13:55:29.527942653 (598h26m29.399025688s) [upload=524552]: Done starting upload
Dec 30 13:55:29.528082490 (598h26m29.399165535s) [upload=524552]: Uploading segments...
Dec 30 13:55:29.528210924 (598h26m29.39929397s) [upload=524552]: Got next segment. Inline: false
Dec 30 13:55:29.528249157 (598h26m29.399332202s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:55:29.528255750 (598h26m29.399338785s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:55:29.775747671 (598h26m29.646830706s) [seg_pos={0 0}, upload=524552]: Begin upload segment...
Dec 30 13:55:29.775791585 (598h26m29.64687462s) [seg_pos={0 0}, upload=524552]: Done begin upload segment.
Dec 30 13:55:29.775816613 (598h26m29.646899648s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.775871357 (598h26m29.646954392s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.776666852 (598h26m29.647749887s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777101459 (598h26m29.648184494s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777497202 (598h26m29.648580237s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.777927573 (598h26m29.649010618s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778175765 (598h26m29.64925882s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778474233 (598h26m29.649557278s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778595244 (598h26m29.649678279s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.778932897 (598h26m29.650015942s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779032897 (598h26m29.650115943s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779350602 (598h26m29.650433647s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.779797082 (598h26m29.650880117s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780239865 (598h26m29.65132291s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.780740239 (598h26m29.651823284s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:29.781264657 (598h26m29.652347692s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Uploading piece...
Dec 30 13:55:31.494971108 (598h26m31.366054143s) [upload=524552]: Next returned nil segment
Dec 30 13:55:31.494995605 (598h26m31.36607864s) [upload=524552]: Waiting for error group managing segments...
Dec 30 13:55:31.788446429 (598h26m31.659529474s) [noise=true , address=145.79.227.7:4294 , piece_id=FGMDG3ZFC3DS2ILHLZSRFGRAATBXRZPCI66O75BWMUJU7WVM2AJA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.791836153 (598h26m31.662919188s) [noise=true , address=145.79.227.5:4223 , piece_id=MUY2APAVRK2NZMIFFRYIUMTQMXLMYV2XMRED5R33EYJN5WZWJAAA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.798297788 (598h26m31.669380833s) [noise=true , address=145.79.227.21:4473 , piece_id=VHJWHTJ6KXIX3RQJQY3OGIDVKC5BSPSMBGS3T7FLAL7FPYCRGDVQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.801691269 (598h26m31.672774304s) [noise=true , address=145.79.227.28:4726 , piece_id=DGUYA3UOPOHCDD2HL7OMONEFZTTFLWLGNLXGECUIB2E6JZPJQQBQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851643325 (598h26m31.72272636s) [noise=true , address=145.79.227.12:4765 , piece_id=L3NY6GWA7LIRG4VA3SFSFTMNLEKILUHFQXPXC2Q5IXP34HSG3YOA, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.851766319 (598h26m31.722849364s) [noise=true , address=145.79.227.26:4654 , piece_id=5TA7MAN5HGY33VOOR3HCAM4E3FXIJ4ALGWITTNRLAH4ZY2MKDFUQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:31.861568735 (598h26m31.73265177s) [noise=true , address=145.79.227.36:4518 , piece_id=375XGGOZX65CC4XW3VDFZWDDL3NLDI4KQ7HEEGVW57C4BHDBKHXQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.116844929 (598h26m31.987927964s) [noise=true , address=77.90.31.12:4590 , piece_id=KBI4WP2NELS7K64X5KWWNMI5MCR6AGBGI4S6RD4OCCWEA7LK7HDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.187932641 (598h26m32.059015676s) [noise=true , address=77.90.31.10:4520 , piece_id=M77SYG6QOU6YTPGVWR3ZI7IIWKERQUFCKLCEZD6YFVHTE7HCPN3Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.209901162 (598h26m32.080984208s) [noise=true , address=45.91.116.12:4228 , piece_id=P54DX5U72EUHZOPQJNRRD6PT5USMT5BAVK5ZXAEU7FICEETLJZ7Q, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.224473169 (598h26m32.095556205s) [noise=true , address=45.91.116.8:4045 , piece_id=VDBWAZ4IHTRM4XEVIJN5L6LXEGJZUEI4FC3GCHKOMXWVA5RNFDDQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.231189418 (598h26m32.102272453s) [noise=true , address=45.91.116.4:4940 , piece_id=S5LDIB3IA2DEBUXN42NF3SIFI6CE72QM6I4DOY6XOCO6LDD4BCIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270421372 (598h26m32.141504417s) [noise=true , address=45.91.116.9:4261 , piece_id=YQRDL57SCQCOKDV3N34FUHZMGUL3LF43FRHLEDSK3Z6SKD575H2A, seg_pos={0 0}, upload=524552]: Done uploading piece. err: <nil>
Dec 30 13:55:32.270444706 (598h26m32.141527741s) [seg_pos={0 0}, upload=524552]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:55:32.270661479 (598h26m32.141744524s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270672410 (598h26m32.141755455s) [noise=true , address=45.91.116.7:4009 , piece_id=II4YBCFS56HEM75RUPUUMVZG3QESRANTGUKY5TXE4CJD2FYAETIQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270709160 (598h26m32.141792205s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 30 13:55:32.270717525 (598h26m32.14180056s) [noise=true , address=45.91.116.5:4299 , piece_id=ZDOGWGWEQ6EXUT226A5RB6G36FHDTITPP7WDB7PPXMNXZEVKWKZQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.270997779 (598h26m32.142080844s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:52186->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 13:55:32.271008560 (598h26m32.142091625s) [noise=true , address=77.90.31.11:4554 , piece_id=R2MASVAGQYYH5ETSC7PKZIR7BNSV73A4QDK7JKVCC5S5WDVNN2AQ, seg_pos={0 0}, upload=524552]: Long tail cancellation due to optimal threshold
Dec 30 13:55:32.271030461 (598h26m32.142113506s) [seg_pos={0 0}, upload=524552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:55:32.271116966 (598h26m32.142200011s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:55:32.271124250 (598h26m32.142207295s) [upload=524552]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:55:32.571519748 (598h26m32.442602793s) [upload=524552]: Done uploading segments...
Dec 30 13:55:32.571542702 (598h26m32.442625737s) [upload=524552]: Upload finished. err: <nil>
Duration: 30.002413087s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [...X...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X.] duration:
29.278382569s trace:
https://jaeger.dev.storj.io/trace/6d8ec002a4742b07
FAIL (2025-12-30 13:55:50 UTC)
orbiter warning: context deadline exceeded
Duration: 29.278406084s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXXX] duration:
5.001023947s trace:
https://jaeger.dev.storj.io/trace/7571848bb69e4490
FAIL (2025-12-30 13:55:11 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001070315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X...X...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.X] duration:
30.001252923s trace:
https://jaeger.dev.storj.io/trace/6d8ec002a4742b07
FAIL (2025-12-30 13:55:06 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001277359s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [................................................X.X..X.XX...] duration:
18.646303858s trace:
https://jaeger.dev.storj.io/trace/5dea0761a983230a
FAIL (2025-12-30 13:55:04 UTC)
orbiter warning: context deadline exceeded
Duration: 18.646350016s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X.......X...X..............X.X.............................] duration:
15.46590148s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:54:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 15.465920356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.............X.XX....XX..XXX.......XXXX.XX.XXX...X...X......] duration:
6.155788261s trace:
https://jaeger.dev.storj.io/trace/539c08e25037f10b
FAIL (2025-12-30 13:54:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
30 13:48:41.423257103 (598h19m41.294340148s) [upload=524454]: Starting upload
Dec 30 13:48:41.423330152 (598h19m41.294413187s) [upload=524454]: Done starting upload
Dec 30 13:48:41.423495918 (598h19m41.294578953s) [upload=524454]: Uploading segments...
Dec 30 13:48:41.423574257 (598h19m41.294657302s) [upload=524454]: Got next segment. Inline: false
Dec 30 13:48:41.423613692 (598h19m41.294696727s) [upload=524454]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:48:41.423619653 (598h19m41.294702698s) [upload=524454]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:48:41.672254842 (598h19m41.543337887s) [seg_pos={0 0}, upload=524454]: Begin upload segment...
Dec 30 13:48:41.672326869 (598h19m41.543409904s) [seg_pos={0 0}, upload=524454]: Done begin upload segment.
Dec 30 13:48:41.672334053 (598h19m41.543417088s) [noise=true , address=145.79.227.20:4439 , piece_id=7MW4GJI6MEM7KYJ3U7UE5YFIEHYQE36YG5SKNZ3ARET36BVZU6FA, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.672376924 (598h19m41.543459969s) [noise=true , address=77.90.31.11:4555 , piece_id=GDMSKDU5CPMF35LX43ENFUE2O4ZEUFHHYTE2OARCJ5NHLQNQFGFQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.672930769 (598h19m41.544013814s) [noise=true , address=145.79.227.6:4257 , piece_id=PBEJ66UFSANVQGTXUB7POF4PJU7CP6RDIRJO6ZAERS4RKWB6AERQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.673398890 (598h19m41.544481925s) [noise=true , address=145.79.227.4:4187 , piece_id=QZE4YQKTYZMG45MUU2XJC63P2JNB3LFAZRT4GHWVDGTH2QG5J5DA, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.673704101 (598h19m41.544787146s) [noise=true , address=45.91.116.11:4117 , piece_id=PIEFAGEMXXXGKWN3NMES34RBVWWNY3CLGBHDMQW32SVXPVJBB2SA, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.673920102 (598h19m41.545003147s) [noise=true , address=145.79.227.21:4473 , piece_id=KTF7KMC5NIZXGR5VEE37XLK5BAOXDGFDBBLZHY7ZCSMXYMRZYYHQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.674268446 (598h19m41.545351481s) [noise=true , address=145.79.227.48:4301 , piece_id=P5RX5OXUMG6ZS7MVKV4DIVVJAAY3YN5A7HEIORC7M5KSKSEEAEWQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.674843711 (598h19m41.545926747s) [noise=true , address=145.79.227.3:4150 , piece_id=7EODFMS5EKJQRRWSCPEFHCI2PVFGHDTZHBRKZP3OOTJY42P2H6HQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.678038906 (598h19m41.549121951s) [noise=true , address=145.79.227.36:4518 , piece_id=AJ2IFGB4N2O6RCHYZW5ZDOPZCBMY5DUBBEITEWGRGJENP47SUWCQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.680001412 (598h19m41.551084448s) [noise=true , address=77.90.31.18:4808 , piece_id=JVRPRAUYTWGADXSP7YDJP7WU4CB2ND2IZXS5A7XSOFY4ZYUS335A, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.686019803 (598h19m41.557102848s) [noise=true , address=145.79.227.37:4566 , piece_id=J3BPX4KQ5QC2PKDH5JOSFYRIZUPTQ56L2NYSCS2PF4RO56WDLMKQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.689270713 (598h19m41.560353758s) [noise=true , address=77.90.31.12:4590 , piece_id=AWTBX2YVNG6E3KLNPISOSFFMXGQ6VXPH2WCCFDEGEKMGVX6SKOPQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.695988444 (598h19m41.567071499s) [noise=true , address=145.79.227.17:4970 , piece_id=JUMESTICWMYGDPCOZJOYIRAH67MRKCCID7HDI3MDCQHNZYNJICNQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.697969176 (598h19m41.569052221s) [noise=true , address=77.90.31.15:4700 , piece_id=MZDOE4GMLW3TKZMSFYUITHEZBVTLZL7XI5QNEYAZGDPWZFPKGHHQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.698641687 (598h19m41.569724732s) [noise=true , address=45.91.116.8:4047 , piece_id=3TWS5OIT3PS2YQRZJEME64HW55GBIWEXZAOOTIEZJNUREEHOBABQ, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:41.700336414 (598h19m41.571419459s) [noise=true , address=145.79.227.45:4231 , piece_id=T5MDLJM7QFHC6PIPQNW2I5W4W3ONIASTVYCQZTIKB6EWXH77S3YA, seg_pos={0 0}, upload=524454]: Uploading piece...
Dec 30 13:48:43.361983488 (598h19m43.233066533s) [upload=524454]: Next returned nil segment
Dec 30 13:48:43.362000932 (598h19m43.233083967s) [upload=524454]: Waiting for error group managing segments...
Dec 30 13:48:43.645990440 (598h19m43.517073475s) [noise=true , address=145.79.227.4:4187 , piece_id=QZE4YQKTYZMG45MUU2XJC63P2JNB3LFAZRT4GHWVDGTH2QG5J5DA, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.653759383 (598h19m43.524842448s) [noise=true , address=145.79.227.20:4439 , piece_id=7MW4GJI6MEM7KYJ3U7UE5YFIEHYQE36YG5SKNZ3ARET36BVZU6FA, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.666046810 (598h19m43.537129855s) [noise=true , address=145.79.227.21:4473 , piece_id=KTF7KMC5NIZXGR5VEE37XLK5BAOXDGFDBBLZHY7ZCSMXYMRZYYHQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.666117565 (598h19m43.5372006s) [noise=true , address=145.79.227.6:4257 , piece_id=PBEJ66UFSANVQGTXUB7POF4PJU7CP6RDIRJO6ZAERS4RKWB6AERQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.681314542 (598h19m43.552397587s) [noise=true , address=145.79.227.37:4566 , piece_id=J3BPX4KQ5QC2PKDH5JOSFYRIZUPTQ56L2NYSCS2PF4RO56WDLMKQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.681914375 (598h19m43.55299741s) [noise=true , address=145.79.227.48:4301 , piece_id=P5RX5OXUMG6ZS7MVKV4DIVVJAAY3YN5A7HEIORC7M5KSKSEEAEWQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.686089605 (598h19m43.55717264s) [noise=true , address=145.79.227.3:4150 , piece_id=7EODFMS5EKJQRRWSCPEFHCI2PVFGHDTZHBRKZP3OOTJY42P2H6HQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.707746633 (598h19m43.578829668s) [noise=true , address=145.79.227.17:4970 , piece_id=JUMESTICWMYGDPCOZJOYIRAH67MRKCCID7HDI3MDCQHNZYNJICNQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.713092191 (598h19m43.584175226s) [noise=true , address=145.79.227.36:4518 , piece_id=AJ2IFGB4N2O6RCHYZW5ZDOPZCBMY5DUBBEITEWGRGJENP47SUWCQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.774854707 (598h19m43.645937742s) [noise=true , address=145.79.227.45:4231 , piece_id=T5MDLJM7QFHC6PIPQNW2I5W4W3ONIASTVYCQZTIKB6EWXH77S3YA, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.839513426 (598h19m43.710596472s) [noise=true , address=77.90.31.11:4555 , piece_id=GDMSKDU5CPMF35LX43ENFUE2O4ZEUFHHYTE2OARCJ5NHLQNQFGFQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.881115733 (598h19m43.752198778s) [noise=true , address=77.90.31.12:4590 , piece_id=AWTBX2YVNG6E3KLNPISOSFFMXGQ6VXPH2WCCFDEGEKMGVX6SKOPQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.905603361 (598h19m43.776686396s) [noise=true , address=77.90.31.18:4808 , piece_id=JVRPRAUYTWGADXSP7YDJP7WU4CB2ND2IZXS5A7XSOFY4ZYUS335A, seg_pos={0 0}, upload=524454]: Done uploading piece. err: <nil>
Dec 30 13:48:43.905623690 (598h19m43.776706725s) [seg_pos={0 0}, upload=524454]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:48:43.905942847 (598h19m43.777025882s) [noise=true , address=45.91.116.8:4047 , piece_id=3TWS5OIT3PS2YQRZJEME64HW55GBIWEXZAOOTIEZJNUREEHOBABQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 30 13:48:43.905952305 (598h19m43.77703535s) [noise=true , address=45.91.116.8:4047 , piece_id=3TWS5OIT3PS2YQRZJEME64HW55GBIWEXZAOOTIEZJNUREEHOBABQ, seg_pos={0 0}, upload=524454]: Long tail cancellation due to optimal threshold
Dec 30 13:48:43.906278196 (598h19m43.777361231s) [noise=true , address=77.90.31.15:4700 , piece_id=MZDOE4GMLW3TKZMSFYUITHEZBVTLZL7XI5QNEYAZGDPWZFPKGHHQ, seg_pos={0 0}, upload=524454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:60544->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:60544->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60544->77.90.31.15:4700: use of closed network connection
Dec 30 13:48:43.906285390 (598h19m43.777368435s) [noise=true , address=77.90.31.15:4700 , piece_id=MZDOE4GMLW3TKZMSFYUITHEZBVTLZL7XI5QNEYAZGDPWZFPKGHHQ, seg_pos={0 0}, upload=524454]: Long tail cancellation due to optimal threshold
Dec 30 13:48:43.906368147 (598h19m43.777451182s) [noise=true , address=45.91.116.11:4117 , piece_id=PIEFAGEMXXXGKWN3NMES34RBVWWNY3CLGBHDMQW32SVXPVJBB2SA, seg_pos={0 0}, upload=524454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:48:43.906379429 (598h19m43.777462464s) [noise=true , address=45.91.116.11:4117 , piece_id=PIEFAGEMXXXGKWN3NMES34RBVWWNY3CLGBHDMQW32SVXPVJBB2SA, seg_pos={0 0}, upload=524454]: Long tail cancellation due to optimal threshold
Dec 30 13:48:43.906396691 (598h19m43.777479726s) [seg_pos={0 0}, upload=524454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:48:43.906465062 (598h19m43.777548097s) [upload=524454]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:48:43.906469580 (598h19m43.777552625s) [upload=524454]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:48:44.198752260 (598h19m44.069835316s) [upload=524454]: Done uploading segments...
Dec 30 13:48:44.198773882 (598h19m44.069856917s) [upload=524454]: Upload finished. err: <nil>
Duration: 6.155831352s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................................................X.X..X.XX..] duration:
19.998281499s trace:
https://jaeger.dev.storj.io/trace/5dea0761a983230a
FAIL (2025-12-30 13:54:19 UTC)
orbiter warning: context deadline exceeded
Duration: 19.998296828s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXX.X.XXXXXX.XXX.XXXXXXXXX.XXXXXXXXXXX.XXXXXXXX] duration:
5.001165346s trace:
https://jaeger.dev.storj.io/trace/5d52505017dd4a4c
FAIL (2025-12-30 13:54:17 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001197477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X.......X...X..............X.X............................] duration:
39.348688355s trace:
https://jaeger.dev.storj.io/trace/215247d8c89b6452
FAIL (2025-12-30 13:54:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece...
Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment
Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments...
Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil>
Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection
Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF
Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection
Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold
Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments...
Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil>
Duration: 39.348712972s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X...X...XXX.......X...XX.XX..X.XX.X...X...X...X....X.XX.XX.] duration:
12.231336745s trace:
https://jaeger.dev.storj.io/trace/4ea93f21d56ffb53
FAIL (2025-12-30 13:54:03 UTC)
orbiter warning: context deadline exceeded
Duration: 12.231368846s
edge
gateway
us-select-1
|