Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
19.209534094s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:13:51 UTC)
orbiter warning: context deadline exceeded
Duration: 19.209564713s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X] duration:
5.001175294s trace:
https://jaeger.dev.storj.io/trace/6b03d5a6fd9c353b
FAIL (2025-12-28 16:13:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00119914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.768719976s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:13:21 UTC)
orbiter warning: context deadline exceeded
Duration: 5.768736468s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....XXXXX.] duration:
6.254823289s trace:
https://jaeger.dev.storj.io/trace/ec7752b51cc2b18
FAIL (2025-12-28 16:13:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded
======LOGS======
16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload
Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload
Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments...
Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false
Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment...
Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment.
Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment
Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments...
Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection
Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold
Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold
Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold
Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments...
Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil>
Duration: 6.254851032s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.969795649s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:13:18 UTC)
orbiter warning: context deadline exceeded
Duration: 5.969820636s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
20.991229939s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:13:15 UTC)
orbiter warning: context deadline exceeded
Duration: 20.991259206s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.] duration:
3.474996753s trace:
https://jaeger.dev.storj.io/trace/7fe2ba39e0ea9472
FAIL (2025-12-28 16:13:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.475025939s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X.....................X.............X.] duration:
51.738721868s trace:
https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb
FAIL (2025-12-28 16:13:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment
Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments...
Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces.
Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection
Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection
Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection
Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF
Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments...
Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil>
Duration: 51.738758909s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....XXXXX] duration:
30.003416158s trace:
https://jaeger.dev.storj.io/trace/ec7752b51cc2b18
FAIL (2025-12-28 16:13:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded
======LOGS======
16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload
Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload
Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments...
Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false
Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment...
Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment.
Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece...
Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment
Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments...
Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil>
Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection
Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold
Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold
Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold
Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments...
Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil>
Duration: 30.003440385s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
7.614953314s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:12:51 UTC)
orbiter warning: context deadline exceeded
Duration: 7.614982048s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
11.664662797s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:12:40 UTC)
orbiter warning: context deadline exceeded
Duration: 11.664692254s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
12.274226645s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:12:26 UTC)
orbiter warning: context deadline exceeded
Duration: 12.274262083s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX] duration:
5.001253864s trace:
https://jaeger.dev.storj.io/trace/7fe2ba39e0ea9472
FAIL (2025-12-28 16:12:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001279183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [XX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....XXXX] duration:
30.003001589s trace:
https://jaeger.dev.storj.io/trace/3ac6eaf7892ec451
FAIL (2025-12-28 16:12:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 5K2XSN2YHCDE77LIOBKAVOHCCOVQJIXP4HF7EX6ADEAA4HUI665Q) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: PTGJCSTLCR4FU2VZKYMBYRBZHX3PUVAYB3XDLZAW3QATIF7IPVXQ) context deadline exceeded
======LOGS======
Dec 28 16:11:44.522718173 (552h42m44.393801218s) [upload=484580]: Starting upload
Dec 28 16:11:44.522806491 (552h42m44.393889536s) [upload=484580]: Done starting upload
Dec 28 16:11:44.522876083 (552h42m44.393959118s) [upload=484580]: Uploading segments...
Dec 28 16:11:44.523058300 (552h42m44.394141345s) [upload=484580]: Got next segment. Inline: false
Dec 28 16:11:44.523086173 (552h42m44.394169208s) [upload=484580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:11:44.523092285 (552h42m44.39417533s) [upload=484580]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:11:44.774317939 (552h42m44.645400974s) [seg_pos={0 0}, upload=484580]: Begin upload segment...
Dec 28 16:11:44.774369407 (552h42m44.645452442s) [seg_pos={0 0}, upload=484580]: Done begin upload segment.
Dec 28 16:11:44.774410055 (552h42m44.64549309s) [noise=true , address=145.79.227.13:4802 , piece_id=5VCPOA2SO3UWIQGVXXWRNXAWMRPLDS3OULOUJVASHUU6GCIQMGWA, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.774464389 (552h42m44.645547434s) [noise=true , address=145.79.227.50:4374 , piece_id=CITVMPRUKVYQULXXIWJDYQBB3JCZELBZIT7F4BQ7X7X2TXO33VPQ, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.774942899 (552h42m44.646025934s) [noise=true , address=77.90.31.13:4626 , piece_id=5K2XSN2YHCDE77LIOBKAVOHCCOVQJIXP4HF7EX6ADEAA4HUI665Q, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.775287836 (552h42m44.646370871s) [noise=true , address=145.79.227.5:4223 , piece_id=P5ATF3SFNBDCHXE7FDQ3XUYAE5RGDL7I7LJDI54ZGJSTJMRTI7LQ, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.775663642 (552h42m44.646746677s) [noise=true , address=145.79.227.31:4798 , piece_id=OPV7WRE5LRNB3HM23Y43URMCIXOGBCYPZ4EA4NWXVBPQCPIYBVEA, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.775999131 (552h42m44.647082166s) [noise=true , address=145.79.227.15:4874 , piece_id=MDBHESUPWAZXLH3IVRVWRQDQJ2VJCSYMZ7NF4YEHYUO2VHAEJMCA, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.776297619 (552h42m44.647380654s) [noise=true , address=77.90.31.14:4663 , piece_id=DCJXG426KG2W4XS7NSUDHAAL2G2ZROFC3YU64G6AETNECAN4FXAQ, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.776612969 (552h42m44.647696004s) [noise=true , address=145.79.227.38:4978 , piece_id=37IU5A4WUAOI4GKOHGBBMQWN377SQGTQWESPZ75BLGYUYI6L7IMQ, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.776943900 (552h42m44.648026935s) [noise=true , address=45.91.116.4:4937 , piece_id=T62QGALV5U7AQQ7ZZQGCH4GFIRYBYCYZ5QML4HO7IKTLUOXMSQSA, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.777942862 (552h42m44.649025897s) [noise=true , address=145.79.227.16:4911 , piece_id=YNZNNNZ3RBSQU2A5K4QC35BPFPCJQVNZCHJMS4D62SXYHAI4JF6Q, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.779109553 (552h42m44.650192598s) [noise=true , address=77.90.31.21:4878 , piece_id=PTGJCSTLCR4FU2VZKYMBYRBZHX3PUVAYB3XDLZAW3QATIF7IPVXQ, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.780325629 (552h42m44.651408674s) [noise=true , address=145.79.227.36:4517 , piece_id=EUQUIEWIX5E52TBFISATMHFHCTTNCULZ3HHJDKGT3KQ554U27VMQ, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.781698984 (552h42m44.652782019s) [noise=true , address=145.79.227.37:4565 , piece_id=5C3Y6Y4OJJI2K35GF3C2OIZW53FPKIW2H5WD7J3RRUIOUNERQOUA, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.786526607 (552h42m44.657609642s) [noise=true , address=145.79.227.14:4839 , piece_id=YFD55U4KXZ6U7EIO2QIVP3I42HTCH3YI36QG4GY3DDIVHMWRG45Q, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.788929282 (552h42m44.660012317s) [noise=true , address=145.79.227.51:4411 , piece_id=JNR3Q6WAJIPZINEC5MP6BITGTXCMYE54WMBBBK6V6YUJHB73PQ6Q, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:44.790380465 (552h42m44.6614635s) [noise=true , address=45.91.116.11:4117 , piece_id=JYOENOVJ7UBS3NNF3KXWPXYDJ4K3QLYQQZ2UBYD6WUG6I7FYW4MA, seg_pos={0 0}, upload=484580]: Uploading piece...
Dec 28 16:11:46.507473437 (552h42m46.378556472s) [upload=484580]: Next returned nil segment
Dec 28 16:11:46.507487354 (552h42m46.378570389s) [upload=484580]: Waiting for error group managing segments...
Dec 28 16:11:46.782772763 (552h42m46.653855798s) [noise=true , address=145.79.227.50:4374 , piece_id=CITVMPRUKVYQULXXIWJDYQBB3JCZELBZIT7F4BQ7X7X2TXO33VPQ, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.782955452 (552h42m46.654038497s) [noise=true , address=145.79.227.5:4223 , piece_id=P5ATF3SFNBDCHXE7FDQ3XUYAE5RGDL7I7LJDI54ZGJSTJMRTI7LQ, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.785453728 (552h42m46.656536763s) [noise=true , address=145.79.227.16:4911 , piece_id=YNZNNNZ3RBSQU2A5K4QC35BPFPCJQVNZCHJMS4D62SXYHAI4JF6Q, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.791381516 (552h42m46.662464562s) [noise=true , address=145.79.227.36:4517 , piece_id=EUQUIEWIX5E52TBFISATMHFHCTTNCULZ3HHJDKGT3KQ554U27VMQ, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.804242876 (552h42m46.675325911s) [noise=true , address=145.79.227.31:4798 , piece_id=OPV7WRE5LRNB3HM23Y43URMCIXOGBCYPZ4EA4NWXVBPQCPIYBVEA, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.804926989 (552h42m46.676010024s) [noise=true , address=145.79.227.38:4978 , piece_id=37IU5A4WUAOI4GKOHGBBMQWN377SQGTQWESPZ75BLGYUYI6L7IMQ, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.808033184 (552h42m46.679116229s) [noise=true , address=145.79.227.37:4565 , piece_id=5C3Y6Y4OJJI2K35GF3C2OIZW53FPKIW2H5WD7J3RRUIOUNERQOUA, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.810459955 (552h42m46.681543s) [noise=true , address=145.79.227.15:4874 , piece_id=MDBHESUPWAZXLH3IVRVWRQDQJ2VJCSYMZ7NF4YEHYUO2VHAEJMCA, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.819101990 (552h42m46.690185035s) [noise=true , address=145.79.227.51:4411 , piece_id=JNR3Q6WAJIPZINEC5MP6BITGTXCMYE54WMBBBK6V6YUJHB73PQ6Q, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.823866542 (552h42m46.694949577s) [noise=true , address=145.79.227.14:4839 , piece_id=YFD55U4KXZ6U7EIO2QIVP3I42HTCH3YI36QG4GY3DDIVHMWRG45Q, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.838438691 (552h42m46.709521736s) [noise=true , address=145.79.227.13:4802 , piece_id=5VCPOA2SO3UWIQGVXXWRNXAWMRPLDS3OULOUJVASHUU6GCIQMGWA, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:46.984941387 (552h42m46.856024422s) [noise=true , address=77.90.31.13:4626 , piece_id=5K2XSN2YHCDE77LIOBKAVOHCCOVQJIXP4HF7EX6ADEAA4HUI665Q, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:47.043164122 (552h42m46.914247157s) [noise=true , address=77.90.31.21:4878 , piece_id=PTGJCSTLCR4FU2VZKYMBYRBZHX3PUVAYB3XDLZAW3QATIF7IPVXQ, seg_pos={0 0}, upload=484580]: Done uploading piece. err: <nil>
Dec 28 16:11:47.043189360 (552h42m46.914272406s) [seg_pos={0 0}, upload=484580]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:11:47.043418226 (552h42m46.914501261s) [noise=true , address=45.91.116.4:4937 , piece_id=T62QGALV5U7AQQ7ZZQGCH4GFIRYBYCYZ5QML4HO7IKTLUOXMSQSA, seg_pos={0 0}, upload=484580]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 28 16:11:47.043426632 (552h42m46.914509667s) [noise=true , address=45.91.116.4:4937 , piece_id=T62QGALV5U7AQQ7ZZQGCH4GFIRYBYCYZ5QML4HO7IKTLUOXMSQSA, seg_pos={0 0}, upload=484580]: Long tail cancellation due to optimal threshold
Dec 28 16:11:47.043450457 (552h42m46.914533492s) [noise=true , address=77.90.31.14:4663 , piece_id=DCJXG426KG2W4XS7NSUDHAAL2G2ZROFC3YU64G6AETNECAN4FXAQ, seg_pos={0 0}, upload=484580]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 28 16:11:47.043459305 (552h42m46.91454234s) [noise=true , address=77.90.31.14:4663 , piece_id=DCJXG426KG2W4XS7NSUDHAAL2G2ZROFC3YU64G6AETNECAN4FXAQ, seg_pos={0 0}, upload=484580]: Long tail cancellation due to optimal threshold
Dec 28 16:11:47.043596125 (552h42m46.91467917s) [noise=true , address=45.91.116.11:4117 , piece_id=JYOENOVJ7UBS3NNF3KXWPXYDJ4K3QLYQQZ2UBYD6WUG6I7FYW4MA, seg_pos={0 0}, upload=484580]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 28 16:11:47.043602086 (552h42m46.914685122s) [noise=true , address=45.91.116.11:4117 , piece_id=JYOENOVJ7UBS3NNF3KXWPXYDJ4K3QLYQQZ2UBYD6WUG6I7FYW4MA, seg_pos={0 0}, upload=484580]: Long tail cancellation due to optimal threshold
Dec 28 16:11:47.043621202 (552h42m46.914704237s) [seg_pos={0 0}, upload=484580]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:11:47.043700223 (552h42m46.914783258s) [upload=484580]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:11:47.043704942 (552h42m46.914787977s) [upload=484580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:11:47.342590169 (552h42m47.213673215s) [upload=484580]: Done uploading segments...
Dec 28 16:11:47.342610718 (552h42m47.213693763s) [upload=484580]: Upload finished. err: <nil>
Duration: 30.003024472s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.....................X.............X] duration:
1m0.00265401s trace:
https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb
FAIL (2025-12-28 16:12:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece...
Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment
Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments...
Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil>
Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces.
Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection
Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection
Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection
Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF
Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold
Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments...
Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil>
Duration: 1m0.002679579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
18.398166291s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:12:07 UTC)
orbiter warning: context deadline exceeded
Duration: 18.398193043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.806637166s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:11:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.806663035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
11.814041049s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 11.814069343s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [XXX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....XXX] duration:
30.003301413s trace:
https://jaeger.dev.storj.io/trace/21dc37b40c601d6d
FAIL (2025-12-28 16:11:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: EVUYDR6SH3EEHYZKAW3MX7ENAM2MJHJZNNL4PYF7ZYKL4OMKWFNQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: WYVELKMRFDK4PU76EYHFLLWWPI5PY4XHSFVCGM6T2GTLCETJWVSQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: PM4JMKIJTLT7DYKRQWDZZRX6XMZRUW4AJJWDYZNWWNKB4AYOAXQA) context deadline exceeded
======LOGS======
Dec 28 16:10:49.202447428 (552h41m49.073530473s) [upload=484568]: Starting upload
Dec 28 16:10:49.202508485 (552h41m49.07359153s) [upload=484568]: Done starting upload
Dec 28 16:10:49.202618735 (552h41m49.07370178s) [upload=484568]: Uploading segments...
Dec 28 16:10:49.202723695 (552h41m49.07380674s) [upload=484568]: Got next segment. Inline: false
Dec 28 16:10:49.202853712 (552h41m49.073936757s) [upload=484568]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:10:49.202862870 (552h41m49.073945915s) [upload=484568]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:10:49.449534103 (552h41m49.320617138s) [seg_pos={0 0}, upload=484568]: Begin upload segment...
Dec 28 16:10:49.449599528 (552h41m49.320682563s) [seg_pos={0 0}, upload=484568]: Done begin upload segment.
Dec 28 16:10:49.449626319 (552h41m49.320709364s) [noise=true , address=145.79.227.37:4567 , piece_id=4N6IWHYA7RMZZJGAOOUVFOUZS2KQ4K64ZIMWDVFZXQUXQMF4VLAQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.449634214 (552h41m49.320717249s) [noise=true , address=145.79.227.8:4330 , piece_id=IWC2KEEYL333IINHIMY4RIMRVKZJOLNSUD6OZXGCMJFBUHEPATOQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.449705009 (552h41m49.320788054s) [noise=true , address=145.79.227.14:4840 , piece_id=ZVOMATMLQLFVXJWJ3WFHSVCBBSL465HPZCONW4CP22HVREDOA36A, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.450352671 (552h41m49.321435706s) [noise=true , address=77.90.31.12:4589 , piece_id=PM4JMKIJTLT7DYKRQWDZZRX6XMZRUW4AJJWDYZNWWNKB4AYOAXQA, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.450725301 (552h41m49.321808346s) [noise=true , address=145.79.227.5:4221 , piece_id=MC25URWFZFJEGV4GNKTW5MLI7LIDO22SAH2THQMO54H5RINTSFLQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.451090507 (552h41m49.322173562s) [noise=true , address=145.79.227.51:4410 , piece_id=RJS4I4NC2SCZE2IFDRYZRXB5VZLKLGGKRJD4X2NDDO37YNM64XNA, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.451470771 (552h41m49.322553806s) [noise=true , address=145.79.227.36:4519 , piece_id=7XPIWBQBSTB4AK3AN23VJV37V7ZWCSO5545YJC3RWA6QB7QIAD2A, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.451872045 (552h41m49.32295509s) [noise=true , address=145.79.227.7:4295 , piece_id=RJ25ZFV73BSUSSDBNEFYBJS33JY3PT5NIAXTR5NA4HPPUBD2G5MQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.452248882 (552h41m49.323331928s) [noise=true , address=45.91.116.13:4156 , piece_id=AVYEMAQM32GIVDIGPL3TQJTU7IQZKSUYIYROOR6NS3OGBDHW6JGA, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.453488674 (552h41m49.324571719s) [noise=true , address=77.90.31.17:4770 , piece_id=EVUYDR6SH3EEHYZKAW3MX7ENAM2MJHJZNNL4PYF7ZYKL4OMKWFNQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.453515885 (552h41m49.32459892s) [noise=true , address=77.90.31.14:4662 , piece_id=Z6E4UOB2PEK6LSSSXQHHQEI5YX5BHJR44P53T7PV6MMBNJCZ4OLA, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.454833574 (552h41m49.325916609s) [noise=true , address=77.90.31.10:4517 , piece_id=WYVELKMRFDK4PU76EYHFLLWWPI5PY4XHSFVCGM6T2GTLCETJWVSQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.455249977 (552h41m49.326333022s) [noise=true , address=45.91.116.5:4297 , piece_id=LCSVQNFQJMZC453AIRUO4KITTRXY4R5SATXA7X6JT2AKQ3S5BBIQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.456924095 (552h41m49.32800713s) [noise=true , address=145.79.227.2:4114 , piece_id=AM4KVXVZN263V4BWCPBVSB3BSBPIEZRNEXY2PUXVMNDNDDWP7ZTA, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.460648216 (552h41m49.331731261s) [noise=true , address=145.79.227.16:4912 , piece_id=JKVZNF5WFTNG7YCX7OTBXJCUAJNLPPQ2J2X3T52UIKUVAKFJ7UTA, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:49.462431892 (552h41m49.333514927s) [noise=true , address=145.79.227.52:4445 , piece_id=CNM5QVT4MVY7LQ3OVDBTFUL4PMUF3JIXI5FNBNAAWKM4UV7PELIQ, seg_pos={0 0}, upload=484568]: Uploading piece...
Dec 28 16:10:51.160745096 (552h41m51.031828141s) [upload=484568]: Next returned nil segment
Dec 28 16:10:51.160768900 (552h41m51.031851945s) [upload=484568]: Waiting for error group managing segments...
Dec 28 16:10:51.448778479 (552h41m51.319861514s) [noise=true , address=145.79.227.2:4114 , piece_id=AM4KVXVZN263V4BWCPBVSB3BSBPIEZRNEXY2PUXVMNDNDDWP7ZTA, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.450519404 (552h41m51.321602449s) [noise=true , address=145.79.227.51:4410 , piece_id=RJS4I4NC2SCZE2IFDRYZRXB5VZLKLGGKRJD4X2NDDO37YNM64XNA, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.456986679 (552h41m51.328069724s) [noise=true , address=145.79.227.7:4295 , piece_id=RJ25ZFV73BSUSSDBNEFYBJS33JY3PT5NIAXTR5NA4HPPUBD2G5MQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.463036148 (552h41m51.334119183s) [noise=true , address=145.79.227.5:4221 , piece_id=MC25URWFZFJEGV4GNKTW5MLI7LIDO22SAH2THQMO54H5RINTSFLQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.470829919 (552h41m51.341912954s) [noise=true , address=145.79.227.36:4519 , piece_id=7XPIWBQBSTB4AK3AN23VJV37V7ZWCSO5545YJC3RWA6QB7QIAD2A, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.475946622 (552h41m51.347029657s) [noise=true , address=145.79.227.14:4840 , piece_id=ZVOMATMLQLFVXJWJ3WFHSVCBBSL465HPZCONW4CP22HVREDOA36A, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.481693124 (552h41m51.352776159s) [noise=true , address=145.79.227.37:4567 , piece_id=4N6IWHYA7RMZZJGAOOUVFOUZS2KQ4K64ZIMWDVFZXQUXQMF4VLAQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.488944352 (552h41m51.360027387s) [noise=true , address=145.79.227.8:4330 , piece_id=IWC2KEEYL333IINHIMY4RIMRVKZJOLNSUD6OZXGCMJFBUHEPATOQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.493707952 (552h41m51.364790988s) [noise=true , address=145.79.227.52:4445 , piece_id=CNM5QVT4MVY7LQ3OVDBTFUL4PMUF3JIXI5FNBNAAWKM4UV7PELIQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.659479209 (552h41m51.530562254s) [noise=true , address=77.90.31.14:4662 , piece_id=Z6E4UOB2PEK6LSSSXQHHQEI5YX5BHJR44P53T7PV6MMBNJCZ4OLA, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.663938800 (552h41m51.535021835s) [noise=true , address=77.90.31.17:4770 , piece_id=EVUYDR6SH3EEHYZKAW3MX7ENAM2MJHJZNNL4PYF7ZYKL4OMKWFNQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:51.754130279 (552h41m51.625213315s) [noise=true , address=77.90.31.10:4517 , piece_id=WYVELKMRFDK4PU76EYHFLLWWPI5PY4XHSFVCGM6T2GTLCETJWVSQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:52.433564216 (552h41m52.304647261s) [noise=true , address=77.90.31.12:4589 , piece_id=PM4JMKIJTLT7DYKRQWDZZRX6XMZRUW4AJJWDYZNWWNKB4AYOAXQA, seg_pos={0 0}, upload=484568]: Done uploading piece. err: <nil>
Dec 28 16:10:52.433589685 (552h41m52.30467273s) [seg_pos={0 0}, upload=484568]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:10:52.433858386 (552h41m52.304941421s) [noise=true , address=45.91.116.5:4297 , piece_id=LCSVQNFQJMZC453AIRUO4KITTRXY4R5SATXA7X6JT2AKQ3S5BBIQ, seg_pos={0 0}, upload=484568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 16:10:52.433868696 (552h41m52.304951731s) [noise=true , address=45.91.116.5:4297 , piece_id=LCSVQNFQJMZC453AIRUO4KITTRXY4R5SATXA7X6JT2AKQ3S5BBIQ, seg_pos={0 0}, upload=484568]: Long tail cancellation due to optimal threshold
Dec 28 16:10:52.433987802 (552h41m52.305070848s) [noise=true , address=145.79.227.16:4912 , piece_id=JKVZNF5WFTNG7YCX7OTBXJCUAJNLPPQ2J2X3T52UIKUVAKFJ7UTA, seg_pos={0 0}, upload=484568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 28 16:10:52.433994295 (552h41m52.30507734s) [noise=true , address=145.79.227.16:4912 , piece_id=JKVZNF5WFTNG7YCX7OTBXJCUAJNLPPQ2J2X3T52UIKUVAKFJ7UTA, seg_pos={0 0}, upload=484568]: Long tail cancellation due to optimal threshold
Dec 28 16:10:52.434011708 (552h41m52.305094743s) [noise=true , address=45.91.116.13:4156 , piece_id=AVYEMAQM32GIVDIGPL3TQJTU7IQZKSUYIYROOR6NS3OGBDHW6JGA, seg_pos={0 0}, upload=484568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 28 16:10:52.434016377 (552h41m52.305099422s) [noise=true , address=45.91.116.13:4156 , piece_id=AVYEMAQM32GIVDIGPL3TQJTU7IQZKSUYIYROOR6NS3OGBDHW6JGA, seg_pos={0 0}, upload=484568]: Long tail cancellation due to optimal threshold
Dec 28 16:10:52.434032878 (552h41m52.305115913s) [seg_pos={0 0}, upload=484568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:10:52.434113462 (552h41m52.305196497s) [upload=484568]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:10:52.434119073 (552h41m52.305202108s) [upload=484568]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:10:52.732819879 (552h41m52.603902924s) [upload=484568]: Done uploading segments...
Dec 28 16:10:52.732842292 (552h41m52.603925337s) [upload=484568]: Upload finished. err: <nil>
Duration: 30.003326911s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [X.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXX] duration:
5.001064391s trace:
https://jaeger.dev.storj.io/trace/665910cdee9285d8
FAIL (2025-12-28 16:11:11 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001086284s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X.....................X.............] duration:
43.080491718s trace:
https://jaeger.dev.storj.io/trace/556a3fb5861f29f
FAIL (2025-12-28 16:10:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
e=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574577548 (552h29m56.445660593s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574886507 (552h29m56.445969542s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575168363 (552h29m56.446251398s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575547866 (552h29m56.446630911s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575844602 (552h29m56.446927647s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576175541 (552h29m56.447258576s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576516070 (552h29m56.447599105s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.578534654 (552h29m56.449617699s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.580192551 (552h29m56.451275596s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.581999100 (552h29m56.453082145s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.584764987 (552h29m56.455848022s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.586884734 (552h29m56.457967799s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.588442480 (552h29m56.459525525s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.590653229 (552h29m56.461736265s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.592369818 (552h29m56.463452853s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:57.859277503 (552h29m57.730360538s) [upload=484402]: Next returned nil segment
Dec 28 15:58:57.859315846 (552h29m57.730398891s) [upload=484402]: Waiting for error group managing segments...
Dec 28 15:58:58.228560330 (552h29m58.099643365s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.252716918 (552h29m58.123799953s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.284245663 (552h29m58.155328698s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.517657659 (552h29m58.388740694s) [noise=true , address=64.49.13.137:4501 , piece_id=MOSBMHVIRHGDLOF6UHCEQJLPUHOPKNOT67X5U4EUG2NDQDYAI65Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.677500887 (552h29m58.548583922s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.712891747 (552h29m58.583974792s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.885850299 (552h29m58.756933344s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.312640901 (552h29m59.183723946s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.375303962 (552h29m59.246386997s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.424009806 (552h29m59.295092851s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.442462813 (552h29m59.313545858s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.446628285 (552h29m59.31771132s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.568135996 (552h29m59.439219031s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766737718 (552h29m59.637820763s) [noise=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766764528 (552h29m59.637847563s) [seg_pos={0 0}, upload=484402]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:58:59.767019034 (552h29m59.638102069s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767029423 (552h29m59.638112458s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767152658 (552h29m59.638235693s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection
Dec 28 15:58:59.767172856 (552h29m59.638255901s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767335827 (552h29m59.638418862s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767345114 (552h29m59.638428159s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767494889 (552h29m59.638577934s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767505590 (552h29m59.638588625s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767542280 (552h29m59.638625325s) [seg_pos={0 0}, upload=484402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:58:59.767650426 (552h29m59.638733481s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:58:59.767658240 (552h29m59.638741275s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:59:00.056943649 (552h29m59.928026694s) [upload=484402]: Done uploading segments...
Dec 28 15:59:00.056965510 (552h29m59.928048555s) [upload=484402]: Upload finished. err: <nil>
Duration: 43.080514321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXX] duration:
5.001002796s trace:
https://jaeger.dev.storj.io/trace/d3e871d873232ae
FAIL (2025-12-28 16:10:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001029698s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
12.495883289s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:10:29 UTC)
orbiter warning: context deadline exceeded
Duration: 12.495912645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
15.525556224s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:10:18 UTC)
orbiter warning: context deadline exceeded
Duration: 15.525655853s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.XXX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....XX] duration:
30.003400317s trace:
https://jaeger.dev.storj.io/trace/5c955d5a15348c01
FAIL (2025-12-28 16:10:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 16:09:46.587694606 (552h40m46.458777651s) [upload=484554]: Starting upload
Dec 28 16:09:46.587796641 (552h40m46.458879676s) [upload=484554]: Done starting upload
Dec 28 16:09:46.587857857 (552h40m46.458940903s) [upload=484554]: Uploading segments...
Dec 28 16:09:46.587991983 (552h40m46.459075018s) [upload=484554]: Got next segment. Inline: false
Dec 28 16:09:46.588028763 (552h40m46.459111798s) [upload=484554]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:09:46.588033843 (552h40m46.459116878s) [upload=484554]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:09:46.836367261 (552h40m46.707450327s) [seg_pos={0 0}, upload=484554]: Begin upload segment...
Dec 28 16:09:46.836445490 (552h40m46.707528535s) [seg_pos={0 0}, upload=484554]: Done begin upload segment.
Dec 28 16:09:46.836458154 (552h40m46.707541189s) [noise=true , address=145.79.227.19:4077 , piece_id=GQNIVKOR7ISD24LGHIV4KCRJXDCUPZG4U5MS342XQEUL7N3E3YNA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.836482781 (552h40m46.707565816s) [noise=true , address=77.90.31.19:4841 , piece_id=LMPCSHRQ4L4GL65TN5F637SKPVUTQNBWLNUXQ6PTMYC6K2DALM5A, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.836855480 (552h40m46.707938515s) [noise=true , address=145.79.227.15:4874 , piece_id=IOTVD37ZZS3KI3LLXQIAWJKEEANOKKQXTTP5QZEIAT6ACMEUY23A, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.837133560 (552h40m46.708216595s) [noise=true , address=145.79.227.4:4187 , piece_id=272UWF5DIKALY2JYJHKG2P53IW75RK2AY7WCYYL2CEK3REX77VKQ, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.837418393 (552h40m46.708501428s) [noise=true , address=145.79.227.7:4295 , piece_id=I76JABQMRP37MFAAZ75ZDDWYIDG5644XVOKJFVXNCXN5ETAV25SA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.837759802 (552h40m46.708842837s) [noise=true , address=145.79.227.48:4302 , piece_id=DRPDJHQ4MXBOFI23U6UF72DQUU2WWPGAVFT4SHBSFEXFZZXSCXGQ, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.838293118 (552h40m46.709376153s) [noise=true , address=45.91.116.9:4261 , piece_id=6PFCOI5ZIV7R6ATLUO7O2L2KI54M23AQLRHPCV5ICIWFVDRZ2IEA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.838714861 (552h40m46.709797896s) [noise=true , address=145.79.227.50:4373 , piece_id=2Q5I6ADNUCCCQR44T56UOVBIZU3YSTHBEVJEX4QIDEHT5EKDIJYA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.839981302 (552h40m46.711064337s) [noise=true , address=145.79.227.6:4258 , piece_id=NMJYBA3A45EMJZERHAVWVPR4ITZ3V3YJDZFBEBRLV2E73C5MLO5A, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.840120047 (552h40m46.711203082s) [noise=true , address=145.79.227.42:4123 , piece_id=OBYZVUQR76DSW2YGKZBU4UQGIQJVBFQ7Z7AUGGN6U2WZMRG2A66A, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.841227896 (552h40m46.712310941s) [noise=true , address=145.79.227.11:4730 , piece_id=H4N7RTXX3QWXZ33MBCCPLLRBPYSCZOWSQTR5OZ2EBYA5H653XTRA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.842533021 (552h40m46.713616056s) [noise=true , address=45.91.116.7:4011 , piece_id=XLFC3GOK3OEA2BNKDITUOBMVKNPHEKFTEKZLEGNFEOZZHGMQX53Q, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.843528637 (552h40m46.714611672s) [noise=true , address=145.79.227.36:4517 , piece_id=7ZOSP7CZSLO6DUFGLHRVPDTX5CANLUNF6BIY6FYOVCIWA4KSHIHQ, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.844687363 (552h40m46.715770398s) [noise=true , address=77.90.31.11:4555 , piece_id=K37DJ3LBRDH5UCAR4ZTGOQFJO4DUDIA6N6E45J5UN65YALZEOESQ, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.845757371 (552h40m46.716840406s) [noise=true , address=145.79.227.2:4113 , piece_id=S7JPG3VJTAXZUMWTCI5UQZ2I43OI27267FXOGDC5QIHXY6I4CJBA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:46.846712039 (552h40m46.717795075s) [noise=true , address=45.91.116.12:4226 , piece_id=BWR2EBU4XUG7KSPENDO3WIOFHGLDIAJSMD77B4QAUL4OE2OMWZNA, seg_pos={0 0}, upload=484554]: Uploading piece...
Dec 28 16:09:48.553380943 (552h40m48.424463988s) [upload=484554]: Next returned nil segment
Dec 28 16:09:48.553407382 (552h40m48.424490427s) [upload=484554]: Waiting for error group managing segments...
Dec 28 16:09:48.850380259 (552h40m48.721463304s) [noise=true , address=145.79.227.50:4373 , piece_id=2Q5I6ADNUCCCQR44T56UOVBIZU3YSTHBEVJEX4QIDEHT5EKDIJYA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.850919064 (552h40m48.722002109s) [noise=true , address=145.79.227.4:4187 , piece_id=272UWF5DIKALY2JYJHKG2P53IW75RK2AY7WCYYL2CEK3REX77VKQ, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.851312714 (552h40m48.722395749s) [noise=true , address=145.79.227.2:4113 , piece_id=S7JPG3VJTAXZUMWTCI5UQZ2I43OI27267FXOGDC5QIHXY6I4CJBA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.851469042 (552h40m48.722552087s) [noise=true , address=145.79.227.6:4258 , piece_id=NMJYBA3A45EMJZERHAVWVPR4ITZ3V3YJDZFBEBRLV2E73C5MLO5A, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.852821587 (552h40m48.723904622s) [noise=true , address=145.79.227.11:4730 , piece_id=H4N7RTXX3QWXZ33MBCCPLLRBPYSCZOWSQTR5OZ2EBYA5H653XTRA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.855891122 (552h40m48.726974167s) [noise=true , address=145.79.227.7:4295 , piece_id=I76JABQMRP37MFAAZ75ZDDWYIDG5644XVOKJFVXNCXN5ETAV25SA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.864846094 (552h40m48.735929139s) [noise=true , address=145.79.227.19:4077 , piece_id=GQNIVKOR7ISD24LGHIV4KCRJXDCUPZG4U5MS342XQEUL7N3E3YNA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.866155907 (552h40m48.737238952s) [noise=true , address=145.79.227.42:4123 , piece_id=OBYZVUQR76DSW2YGKZBU4UQGIQJVBFQ7Z7AUGGN6U2WZMRG2A66A, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.868053782 (552h40m48.739136817s) [noise=true , address=145.79.227.36:4517 , piece_id=7ZOSP7CZSLO6DUFGLHRVPDTX5CANLUNF6BIY6FYOVCIWA4KSHIHQ, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.868249885 (552h40m48.73933292s) [noise=true , address=145.79.227.48:4302 , piece_id=DRPDJHQ4MXBOFI23U6UF72DQUU2WWPGAVFT4SHBSFEXFZZXSCXGQ, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:48.868778862 (552h40m48.739861897s) [noise=true , address=145.79.227.15:4874 , piece_id=IOTVD37ZZS3KI3LLXQIAWJKEEANOKKQXTTP5QZEIAT6ACMEUY23A, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:49.017070995 (552h40m48.88815403s) [noise=true , address=77.90.31.19:4841 , piece_id=LMPCSHRQ4L4GL65TN5F637SKPVUTQNBWLNUXQ6PTMYC6K2DALM5A, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:49.154453714 (552h40m49.025536759s) [noise=true , address=45.91.116.9:4261 , piece_id=6PFCOI5ZIV7R6ATLUO7O2L2KI54M23AQLRHPCV5ICIWFVDRZ2IEA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: <nil>
Dec 28 16:09:49.154481036 (552h40m49.025564071s) [seg_pos={0 0}, upload=484554]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:09:49.154721985 (552h40m49.02580503s) [noise=true , address=45.91.116.7:4011 , piece_id=XLFC3GOK3OEA2BNKDITUOBMVKNPHEKFTEKZLEGNFEOZZHGMQX53Q, seg_pos={0 0}, upload=484554]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 28 16:09:49.154732927 (552h40m49.025815972s) [noise=true , address=45.91.116.7:4011 , piece_id=XLFC3GOK3OEA2BNKDITUOBMVKNPHEKFTEKZLEGNFEOZZHGMQX53Q, seg_pos={0 0}, upload=484554]: Long tail cancellation due to optimal threshold
Dec 28 16:09:49.155121076 (552h40m49.026204121s) [noise=true , address=77.90.31.11:4555 , piece_id=K37DJ3LBRDH5UCAR4ZTGOQFJO4DUDIA6N6E45J5UN65YALZEOESQ, seg_pos={0 0}, upload=484554]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 28 16:09:49.155131425 (552h40m49.02621447s) [noise=true , address=77.90.31.11:4555 , piece_id=K37DJ3LBRDH5UCAR4ZTGOQFJO4DUDIA6N6E45J5UN65YALZEOESQ, seg_pos={0 0}, upload=484554]: Long tail cancellation due to optimal threshold
Dec 28 16:09:49.155306969 (552h40m49.026390014s) [noise=true , address=45.91.116.12:4226 , piece_id=BWR2EBU4XUG7KSPENDO3WIOFHGLDIAJSMD77B4QAUL4OE2OMWZNA, seg_pos={0 0}, upload=484554]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 28 16:09:49.155316708 (552h40m49.026399773s) [noise=true , address=45.91.116.12:4226 , piece_id=BWR2EBU4XUG7KSPENDO3WIOFHGLDIAJSMD77B4QAUL4OE2OMWZNA, seg_pos={0 0}, upload=484554]: Long tail cancellation due to optimal threshold
Dec 28 16:09:49.155339822 (552h40m49.026422857s) [seg_pos={0 0}, upload=484554]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:09:49.155425174 (552h40m49.026508219s) [upload=484554]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:09:49.155432459 (552h40m49.026515504s) [upload=484554]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:09:49.457465850 (552h40m49.328548895s) [upload=484554]: Done uploading segments...
Dec 28 16:09:49.457490707 (552h40m49.328573742s) [upload=484554]: Upload finished. err: <nil>
Duration: 30.003413933s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XXX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XX] duration:
5.000879089s trace:
https://jaeger.dev.storj.io/trace/f4aee05e27bb5f7
FAIL (2025-12-28 16:09:58 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000930718s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [X........................X.....................X............] duration:
48.026178169s trace:
https://jaeger.dev.storj.io/trace/556a3fb5861f29f
FAIL (2025-12-28 16:09:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
e=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574577548 (552h29m56.445660593s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574886507 (552h29m56.445969542s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575168363 (552h29m56.446251398s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575547866 (552h29m56.446630911s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575844602 (552h29m56.446927647s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576175541 (552h29m56.447258576s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576516070 (552h29m56.447599105s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.578534654 (552h29m56.449617699s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.580192551 (552h29m56.451275596s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.581999100 (552h29m56.453082145s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.584764987 (552h29m56.455848022s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.586884734 (552h29m56.457967799s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.588442480 (552h29m56.459525525s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.590653229 (552h29m56.461736265s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.592369818 (552h29m56.463452853s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:57.859277503 (552h29m57.730360538s) [upload=484402]: Next returned nil segment
Dec 28 15:58:57.859315846 (552h29m57.730398891s) [upload=484402]: Waiting for error group managing segments...
Dec 28 15:58:58.228560330 (552h29m58.099643365s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.252716918 (552h29m58.123799953s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.284245663 (552h29m58.155328698s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.517657659 (552h29m58.388740694s) [noise=true , address=64.49.13.137:4501 , piece_id=MOSBMHVIRHGDLOF6UHCEQJLPUHOPKNOT67X5U4EUG2NDQDYAI65Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.677500887 (552h29m58.548583922s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.712891747 (552h29m58.583974792s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.885850299 (552h29m58.756933344s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.312640901 (552h29m59.183723946s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.375303962 (552h29m59.246386997s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.424009806 (552h29m59.295092851s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.442462813 (552h29m59.313545858s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.446628285 (552h29m59.31771132s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.568135996 (552h29m59.439219031s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766737718 (552h29m59.637820763s) [noise=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766764528 (552h29m59.637847563s) [seg_pos={0 0}, upload=484402]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:58:59.767019034 (552h29m59.638102069s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767029423 (552h29m59.638112458s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767152658 (552h29m59.638235693s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection
Dec 28 15:58:59.767172856 (552h29m59.638255901s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767335827 (552h29m59.638418862s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767345114 (552h29m59.638428159s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767494889 (552h29m59.638577934s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767505590 (552h29m59.638588625s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767542280 (552h29m59.638625325s) [seg_pos={0 0}, upload=484402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:58:59.767650426 (552h29m59.638733481s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:58:59.767658240 (552h29m59.638741275s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:59:00.056943649 (552h29m59.928026694s) [upload=484402]: Done uploading segments...
Dec 28 15:59:00.056965510 (552h29m59.928048555s) [upload=484402]: Upload finished. err: <nil>
Duration: 48.026209689s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
13.934219757s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:09:55 UTC)
orbiter warning: context deadline exceeded
Duration: 13.934245597s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.XXX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....X] duration:
30.00262837s trace:
https://jaeger.dev.storj.io/trace/843da4a48bdedc0
FAIL (2025-12-28 16:09:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: JRCGOBEFDJ2JQPJQPSQJFED3VW26G4EFC54LU72PVIRISSWQ6HPQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: O5LLTJMKYXISXGS67XH2HB2LRKH4NRC36XWIDDKWXIZVIZ5EYZJQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: UHO5LLE7OXMXX5LKAR57JKNNPLQTYJ2SURDMUB2XTMPLDILZLCVA) context deadline exceeded
======LOGS======
28 16:09:16.370606299 (552h40m16.241689344s) [upload=484549]: Starting upload
Dec 28 16:09:16.370710597 (552h40m16.241793642s) [upload=484549]: Done starting upload
Dec 28 16:09:16.370831037 (552h40m16.241914082s) [upload=484549]: Uploading segments...
Dec 28 16:09:16.371101982 (552h40m16.242185027s) [upload=484549]: Got next segment. Inline: false
Dec 28 16:09:16.371158490 (552h40m16.242241535s) [upload=484549]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:09:16.371166215 (552h40m16.24224926s) [upload=484549]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:09:16.620229293 (552h40m16.491312358s) [seg_pos={0 0}, upload=484549]: Begin upload segment...
Dec 28 16:09:16.620318673 (552h40m16.491401708s) [seg_pos={0 0}, upload=484549]: Done begin upload segment.
Dec 28 16:09:16.620389417 (552h40m16.491472462s) [noise=true , address=145.79.227.36:4517 , piece_id=5QRFB3APSANJBCFW6CKNZACZ2VG2YSKFHRCBLDJJ6X65VJZGVH7Q, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.620440385 (552h40m16.49152342s) [noise=true , address=145.79.227.13:4803 , piece_id=RLYCPKMXE3QS35RTZ3JFTWQFHB3WOILVYIETBFHPOUNCIIJCTJCQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.621122784 (552h40m16.492205829s) [noise=true , address=45.91.116.7:4011 , piece_id=JRCGOBEFDJ2JQPJQPSQJFED3VW26G4EFC54LU72PVIRISSWQ6HPQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.621573793 (552h40m16.492656838s) [noise=true , address=77.90.31.19:4841 , piece_id=O5LLTJMKYXISXGS67XH2HB2LRKH4NRC36XWIDDKWXIZVIZ5EYZJQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.622043257 (552h40m16.493126302s) [noise=true , address=45.91.116.9:4261 , piece_id=JYL33MYPFRBOSEOX6G5SNC4575GFRSAECTYMZQW2FXZGUZQHXWRQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.622061912 (552h40m16.493144947s) [noise=true , address=77.90.31.17:4773 , piece_id=UHO5LLE7OXMXX5LKAR57JKNNPLQTYJ2SURDMUB2XTMPLDILZLCVA, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.622556003 (552h40m16.493639048s) [noise=true , address=145.79.227.16:4912 , piece_id=ML2ZMAU4YUK5PFPYOVXBKGDGMHWASZIZ4LXTRP2Q2JARMMPGXA2A, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.622656825 (552h40m16.49373986s) [noise=true , address=145.79.227.6:4258 , piece_id=VQDUV2C5NDUEYRNM3BJ557M2Z6GLGQSVZHZ2TTHG2YXZ67MARX6Q, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.623005008 (552h40m16.494088053s) [noise=true , address=145.79.227.19:4079 , piece_id=WAC2HVIEJQP7YY4LW6FDDXIUG63TQYGLNTGKQGA3W263VBE7WXVA, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.623031158 (552h40m16.494114193s) [noise=true , address=145.79.227.12:4765 , piece_id=VIV3JDWBKBEPUI3XI3NX42BK4JVEKLPF4XBBYYVDYLZLO5IEPAPQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.623397556 (552h40m16.494480601s) [noise=true , address=145.79.227.14:4839 , piece_id=O2A7YMVAYHHUOTMD6Z2OOJZSUFI7ADH5ZPJGBQFBR4IWBIMZSMYQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.623816093 (552h40m16.494899128s) [noise=true , address=145.79.227.40:4050 , piece_id=GFWNFGWQ2QS64JHQIHH7Q3EMWVWTULHI2BNUFEZ74EGSJBDZEVIQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.623829608 (552h40m16.494912653s) [noise=true , address=145.79.227.48:4302 , piece_id=QPIBMVRVP3AYRLZVBZ2QOQAIMRF7TI4SNZJE2UY577BNFDOLOUIQ, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.624211035 (552h40m16.49529407s) [noise=true , address=145.79.227.18:4007 , piece_id=4FNQWGTCOTLPAV6JISYBG2XAYS4QWWUTIQGDCP667A3BZ7SFDGNA, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.624496137 (552h40m16.495579172s) [noise=true , address=45.91.116.13:4153 , piece_id=UXVJUKO4BCTOHJJCFMNJ6AR2S3XCSIF52K55N3GPZNT4ATQ6TGTA, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:16.624633308 (552h40m16.495716354s) [noise=true , address=145.79.227.51:4409 , piece_id=HTF7MPP6DSH6IXXOJBNERA6B3GZBOSA2RX62IL3N3JRZJLBXWR3A, seg_pos={0 0}, upload=484549]: Uploading piece...
Dec 28 16:09:18.240033801 (552h40m18.111116846s) [upload=484549]: Next returned nil segment
Dec 28 16:09:18.240045634 (552h40m18.111128669s) [upload=484549]: Waiting for error group managing segments...
Dec 28 16:09:18.618751503 (552h40m18.489834538s) [noise=true , address=145.79.227.48:4302 , piece_id=QPIBMVRVP3AYRLZVBZ2QOQAIMRF7TI4SNZJE2UY577BNFDOLOUIQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.620217625 (552h40m18.49130067s) [noise=true , address=145.79.227.12:4765 , piece_id=VIV3JDWBKBEPUI3XI3NX42BK4JVEKLPF4XBBYYVDYLZLO5IEPAPQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.621503964 (552h40m18.492587009s) [noise=true , address=145.79.227.51:4409 , piece_id=HTF7MPP6DSH6IXXOJBNERA6B3GZBOSA2RX62IL3N3JRZJLBXWR3A, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.626522911 (552h40m18.497605946s) [noise=true , address=145.79.227.19:4079 , piece_id=WAC2HVIEJQP7YY4LW6FDDXIUG63TQYGLNTGKQGA3W263VBE7WXVA, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.626872547 (552h40m18.497955592s) [noise=true , address=145.79.227.13:4803 , piece_id=RLYCPKMXE3QS35RTZ3JFTWQFHB3WOILVYIETBFHPOUNCIIJCTJCQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.633249960 (552h40m18.504332995s) [noise=true , address=145.79.227.18:4007 , piece_id=4FNQWGTCOTLPAV6JISYBG2XAYS4QWWUTIQGDCP667A3BZ7SFDGNA, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.640001927 (552h40m18.511084962s) [noise=true , address=145.79.227.36:4517 , piece_id=5QRFB3APSANJBCFW6CKNZACZ2VG2YSKFHRCBLDJJ6X65VJZGVH7Q, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.642558095 (552h40m18.51364113s) [noise=true , address=145.79.227.6:4258 , piece_id=VQDUV2C5NDUEYRNM3BJ557M2Z6GLGQSVZHZ2TTHG2YXZ67MARX6Q, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.665967150 (552h40m18.537050195s) [noise=true , address=145.79.227.14:4839 , piece_id=O2A7YMVAYHHUOTMD6Z2OOJZSUFI7ADH5ZPJGBQFBR4IWBIMZSMYQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:18.921951707 (552h40m18.793034752s) [noise=true , address=77.90.31.17:4773 , piece_id=UHO5LLE7OXMXX5LKAR57JKNNPLQTYJ2SURDMUB2XTMPLDILZLCVA, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:19.034315817 (552h40m18.905398862s) [noise=true , address=45.91.116.9:4261 , piece_id=JYL33MYPFRBOSEOX6G5SNC4575GFRSAECTYMZQW2FXZGUZQHXWRQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:19.515998521 (552h40m19.387081557s) [noise=true , address=77.90.31.19:4841 , piece_id=O5LLTJMKYXISXGS67XH2HB2LRKH4NRC36XWIDDKWXIZVIZ5EYZJQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:19.743510452 (552h40m19.614593497s) [noise=true , address=45.91.116.7:4011 , piece_id=JRCGOBEFDJ2JQPJQPSQJFED3VW26G4EFC54LU72PVIRISSWQ6HPQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: <nil>
Dec 28 16:09:19.743536822 (552h40m19.614619867s) [seg_pos={0 0}, upload=484549]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:09:19.743753665 (552h40m19.61483672s) [noise=true , address=145.79.227.16:4912 , piece_id=ML2ZMAU4YUK5PFPYOVXBKGDGMHWASZIZ4LXTRP2Q2JARMMPGXA2A, seg_pos={0 0}, upload=484549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:59606->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:59606->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59606->145.79.227.16:4912: use of closed network connection
Dec 28 16:09:19.743762553 (552h40m19.614845588s) [noise=true , address=145.79.227.16:4912 , piece_id=ML2ZMAU4YUK5PFPYOVXBKGDGMHWASZIZ4LXTRP2Q2JARMMPGXA2A, seg_pos={0 0}, upload=484549]: Long tail cancellation due to optimal threshold
Dec 28 16:09:19.743883502 (552h40m19.614966537s) [noise=true , address=145.79.227.40:4050 , piece_id=GFWNFGWQ2QS64JHQIHH7Q3EMWVWTULHI2BNUFEZ74EGSJBDZEVIQ, seg_pos={0 0}, upload=484549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 28 16:09:19.743897829 (552h40m19.614980874s) [noise=true , address=145.79.227.40:4050 , piece_id=GFWNFGWQ2QS64JHQIHH7Q3EMWVWTULHI2BNUFEZ74EGSJBDZEVIQ, seg_pos={0 0}, upload=484549]: Long tail cancellation due to optimal threshold
Dec 28 16:09:19.744021305 (552h40m19.61510435s) [noise=true , address=45.91.116.13:4153 , piece_id=UXVJUKO4BCTOHJJCFMNJ6AR2S3XCSIF52K55N3GPZNT4ATQ6TGTA, seg_pos={0 0}, upload=484549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 28 16:09:19.744026865 (552h40m19.6151099s) [noise=true , address=45.91.116.13:4153 , piece_id=UXVJUKO4BCTOHJJCFMNJ6AR2S3XCSIF52K55N3GPZNT4ATQ6TGTA, seg_pos={0 0}, upload=484549]: Long tail cancellation due to optimal threshold
Dec 28 16:09:19.744046613 (552h40m19.615129648s) [seg_pos={0 0}, upload=484549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:09:19.744136153 (552h40m19.615219188s) [upload=484549]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:09:19.744141052 (552h40m19.615224087s) [upload=484549]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:09:20.043316062 (552h40m19.914399097s) [upload=484549]: Done uploading segments...
Dec 28 16:09:20.043337442 (552h40m19.914420507s) [upload=484549]: Upload finished. err: <nil>
Duration: 30.002638781s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
12.985281237s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:09:42 UTC)
orbiter warning: context deadline exceeded
Duration: 12.985313099s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [.XXX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..X] duration:
5.000803586s trace:
https://jaeger.dev.storj.io/trace/ffb93bbbb7845fe
FAIL (2025-12-28 16:09:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00082646s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.407420641s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:09:25 UTC)
orbiter warning: context deadline exceeded
Duration: 6.407485935s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
15.684768111s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:09:13 UTC)
orbiter warning: context deadline exceeded
Duration: 15.684799531s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X.XXX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X....] duration:
5.875594123s trace:
https://jaeger.dev.storj.io/trace/19fdb56c6a50b033
FAIL (2025-12-28 16:09:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ) context deadline exceeded
======LOGS======
Dec 28 16:04:37.528485291 (552h35m37.399568326s) [upload=484480]: Starting upload
Dec 28 16:04:37.528577968 (552h35m37.399661013s) [upload=484480]: Done starting upload
Dec 28 16:04:37.528640837 (552h35m37.399723882s) [upload=484480]: Uploading segments...
Dec 28 16:04:37.528978690 (552h35m37.400061735s) [upload=484480]: Got next segment. Inline: false
Dec 28 16:04:37.529013637 (552h35m37.400096682s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:04:37.529019909 (552h35m37.400102964s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:04:37.770378807 (552h35m37.641461842s) [seg_pos={0 0}, upload=484480]: Begin upload segment...
Dec 28 16:04:37.770445314 (552h35m37.641528359s) [seg_pos={0 0}, upload=484480]: Done begin upload segment.
Dec 28 16:04:37.770462446 (552h35m37.641545481s) [noise=true , address=145.79.227.47:4267 , piece_id=UPD34XVZABWACXLWURSGV7ANC5A7H426HE6WRHIYA6ARIYM3K36A, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.770542650 (552h35m37.641625685s) [noise=true , address=145.79.227.31:4798 , piece_id=ZA7N5IBZGUK47NTZFPPJFPY4IPWELZ6GUQ7T653OKI3LTFQ7XDVA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.771212455 (552h35m37.64229549s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.771731523 (552h35m37.642814558s) [noise=true , address=145.79.227.13:4804 , piece_id=GLEEPRVNORAMF6SK6RAMFCCS5NK5ROL3CDEJQVL4VJIW45WYOAHQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.772707151 (552h35m37.643790186s) [noise=true , address=145.79.227.4:4187 , piece_id=NSCA3K7MANN77LRJHXLBXIQGA7NEOR2Z2DYT7YK74KM33JJHWMGQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.772783767 (552h35m37.643866822s) [noise=true , address=77.90.31.13:4629 , piece_id=DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.773174921 (552h35m37.644257956s) [noise=true , address=145.79.227.7:4295 , piece_id=JGDXWWW7AGQYVF5ZL4ZXSCCFDNIEAYBGUOOKJVCQ3YXYA4J2PPSQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.773858233 (552h35m37.644941278s) [noise=true , address=145.79.227.16:4912 , piece_id=TJIAJ26SUAVZXCH23MYH3LCVKIZWU4F74LGF4CPMM5BW5MQOXQ5Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.774456882 (552h35m37.645539928s) [noise=true , address=145.79.227.41:4086 , piece_id=LO2ZSGF7Q2ZT5XAW6FK2Y2OX3V6JLJHETO7FXXMVFCSYFPOPUOMQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.774935053 (552h35m37.646018098s) [noise=true , address=145.79.227.19:4079 , piece_id=OO7QFBONFN2NUHA3ZXDWNXMVJ6X43FPTXSGAIIWHU5Q3DPXLBCCA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.775395209 (552h35m37.646478254s) [noise=true , address=77.90.31.11:4554 , piece_id=GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.775479529 (552h35m37.646562574s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.776129157 (552h35m37.647212232s) [noise=true , address=145.79.227.32:4835 , piece_id=CRBENMVGVYOOPGPXB6CIDJNFWH74LYCC4NZ4IJL5WKJMNRNIBDFA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.776762543 (552h35m37.647845578s) [noise=true , address=145.79.227.3:4150 , piece_id=LSXOVSZ2ATLO33UJZ74GSX6NACVIJE5SBF3XG4DOHTDHKLZTNJRA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.777421568 (552h35m37.648504623s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.777743390 (552h35m37.648826425s) [noise=true , address=145.79.227.27:4691 , piece_id=SRE4NU7DQVYPXB4AGW6WNYC6PGRMKPGKQBY7I7UUMT3IQEVPQ2UQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:39.482070524 (552h35m39.353153559s) [upload=484480]: Next returned nil segment
Dec 28 16:04:39.482082457 (552h35m39.353165492s) [upload=484480]: Waiting for error group managing segments...
Dec 28 16:04:39.766675369 (552h35m39.637758404s) [noise=true , address=145.79.227.31:4798 , piece_id=ZA7N5IBZGUK47NTZFPPJFPY4IPWELZ6GUQ7T653OKI3LTFQ7XDVA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.783697843 (552h35m39.654780888s) [noise=true , address=145.79.227.16:4912 , piece_id=TJIAJ26SUAVZXCH23MYH3LCVKIZWU4F74LGF4CPMM5BW5MQOXQ5Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.790128158 (552h35m39.661211193s) [noise=true , address=145.79.227.27:4691 , piece_id=SRE4NU7DQVYPXB4AGW6WNYC6PGRMKPGKQBY7I7UUMT3IQEVPQ2UQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.791084600 (552h35m39.662167635s) [noise=true , address=145.79.227.47:4267 , piece_id=UPD34XVZABWACXLWURSGV7ANC5A7H426HE6WRHIYA6ARIYM3K36A, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.791916333 (552h35m39.662999379s) [noise=true , address=145.79.227.41:4086 , piece_id=LO2ZSGF7Q2ZT5XAW6FK2Y2OX3V6JLJHETO7FXXMVFCSYFPOPUOMQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.792215172 (552h35m39.663298207s) [noise=true , address=145.79.227.19:4079 , piece_id=OO7QFBONFN2NUHA3ZXDWNXMVJ6X43FPTXSGAIIWHU5Q3DPXLBCCA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.797526476 (552h35m39.668609521s) [noise=true , address=145.79.227.32:4835 , piece_id=CRBENMVGVYOOPGPXB6CIDJNFWH74LYCC4NZ4IJL5WKJMNRNIBDFA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.799022304 (552h35m39.670105349s) [noise=true , address=145.79.227.3:4150 , piece_id=LSXOVSZ2ATLO33UJZ74GSX6NACVIJE5SBF3XG4DOHTDHKLZTNJRA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.801242962 (552h35m39.672326007s) [noise=true , address=145.79.227.7:4295 , piece_id=JGDXWWW7AGQYVF5ZL4ZXSCCFDNIEAYBGUOOKJVCQ3YXYA4J2PPSQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.802436274 (552h35m39.673519319s) [noise=true , address=145.79.227.4:4187 , piece_id=NSCA3K7MANN77LRJHXLBXIQGA7NEOR2Z2DYT7YK74KM33JJHWMGQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.807835044 (552h35m39.678918099s) [noise=true , address=145.79.227.13:4804 , piece_id=GLEEPRVNORAMF6SK6RAMFCCS5NK5ROL3CDEJQVL4VJIW45WYOAHQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.951953701 (552h35m39.823036736s) [noise=true , address=77.90.31.13:4629 , piece_id=DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.969307625 (552h35m39.84039067s) [noise=true , address=77.90.31.11:4554 , piece_id=GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.969326061 (552h35m39.840409096s) [seg_pos={0 0}, upload=484480]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:04:39.969478250 (552h35m39.840561285s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969486115 (552h35m39.84056916s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969707167 (552h35m39.840790202s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969713068 (552h35m39.840796103s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969853796 (552h35m39.840936831s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969859626 (552h35m39.840942662s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969874816 (552h35m39.840957861s) [seg_pos={0 0}, upload=484480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:04:39.969938877 (552h35m39.841021912s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:04:39.969943717 (552h35m39.841026752s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:04:40.259688480 (552h35m40.130771525s) [upload=484480]: Done uploading segments...
Dec 28 16:04:40.259727294 (552h35m40.130810329s) [upload=484480]: Upload finished. err: <nil>
Duration: 5.875619603s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [X.XXX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..] duration:
4.159921276s trace:
https://jaeger.dev.storj.io/trace/21c3b5f5b9af47d4
FAIL (2025-12-28 16:08:52 UTC)
orbiter warning: context deadline exceeded
Duration: 4.159948497s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X........................X.....................X...........] duration:
33.290998664s trace:
https://jaeger.dev.storj.io/trace/556a3fb5861f29f
FAIL (2025-12-28 16:08:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
e=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574577548 (552h29m56.445660593s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574886507 (552h29m56.445969542s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575168363 (552h29m56.446251398s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575547866 (552h29m56.446630911s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575844602 (552h29m56.446927647s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576175541 (552h29m56.447258576s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576516070 (552h29m56.447599105s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.578534654 (552h29m56.449617699s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.580192551 (552h29m56.451275596s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.581999100 (552h29m56.453082145s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.584764987 (552h29m56.455848022s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.586884734 (552h29m56.457967799s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.588442480 (552h29m56.459525525s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.590653229 (552h29m56.461736265s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.592369818 (552h29m56.463452853s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:57.859277503 (552h29m57.730360538s) [upload=484402]: Next returned nil segment
Dec 28 15:58:57.859315846 (552h29m57.730398891s) [upload=484402]: Waiting for error group managing segments...
Dec 28 15:58:58.228560330 (552h29m58.099643365s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.252716918 (552h29m58.123799953s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.284245663 (552h29m58.155328698s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.517657659 (552h29m58.388740694s) [noise=true , address=64.49.13.137:4501 , piece_id=MOSBMHVIRHGDLOF6UHCEQJLPUHOPKNOT67X5U4EUG2NDQDYAI65Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.677500887 (552h29m58.548583922s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.712891747 (552h29m58.583974792s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.885850299 (552h29m58.756933344s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.312640901 (552h29m59.183723946s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.375303962 (552h29m59.246386997s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.424009806 (552h29m59.295092851s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.442462813 (552h29m59.313545858s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.446628285 (552h29m59.31771132s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.568135996 (552h29m59.439219031s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766737718 (552h29m59.637820763s) [noise=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766764528 (552h29m59.637847563s) [seg_pos={0 0}, upload=484402]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:58:59.767019034 (552h29m59.638102069s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767029423 (552h29m59.638112458s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767152658 (552h29m59.638235693s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection
Dec 28 15:58:59.767172856 (552h29m59.638255901s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767335827 (552h29m59.638418862s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767345114 (552h29m59.638428159s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767494889 (552h29m59.638577934s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767505590 (552h29m59.638588625s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767542280 (552h29m59.638625325s) [seg_pos={0 0}, upload=484402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:58:59.767650426 (552h29m59.638733481s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:58:59.767658240 (552h29m59.638741275s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:59:00.056943649 (552h29m59.928026694s) [upload=484402]: Done uploading segments...
Dec 28 15:59:00.056965510 (552h29m59.928048555s) [upload=484402]: Upload finished. err: <nil>
Duration: 33.291014636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
12.096129242s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:08:42 UTC)
orbiter warning: context deadline exceeded
Duration: 12.096155522s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X........................X.....................X..........] duration:
42.362946785s trace:
https://jaeger.dev.storj.io/trace/556a3fb5861f29f
FAIL (2025-12-28 16:08:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
e=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574577548 (552h29m56.445660593s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574886507 (552h29m56.445969542s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575168363 (552h29m56.446251398s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575547866 (552h29m56.446630911s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575844602 (552h29m56.446927647s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576175541 (552h29m56.447258576s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576516070 (552h29m56.447599105s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.578534654 (552h29m56.449617699s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.580192551 (552h29m56.451275596s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.581999100 (552h29m56.453082145s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.584764987 (552h29m56.455848022s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.586884734 (552h29m56.457967799s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.588442480 (552h29m56.459525525s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.590653229 (552h29m56.461736265s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.592369818 (552h29m56.463452853s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:57.859277503 (552h29m57.730360538s) [upload=484402]: Next returned nil segment
Dec 28 15:58:57.859315846 (552h29m57.730398891s) [upload=484402]: Waiting for error group managing segments...
Dec 28 15:58:58.228560330 (552h29m58.099643365s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.252716918 (552h29m58.123799953s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.284245663 (552h29m58.155328698s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.517657659 (552h29m58.388740694s) [noise=true , address=64.49.13.137:4501 , piece_id=MOSBMHVIRHGDLOF6UHCEQJLPUHOPKNOT67X5U4EUG2NDQDYAI65Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.677500887 (552h29m58.548583922s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.712891747 (552h29m58.583974792s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.885850299 (552h29m58.756933344s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.312640901 (552h29m59.183723946s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.375303962 (552h29m59.246386997s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.424009806 (552h29m59.295092851s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.442462813 (552h29m59.313545858s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.446628285 (552h29m59.31771132s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.568135996 (552h29m59.439219031s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766737718 (552h29m59.637820763s) [noise=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766764528 (552h29m59.637847563s) [seg_pos={0 0}, upload=484402]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:58:59.767019034 (552h29m59.638102069s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767029423 (552h29m59.638112458s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767152658 (552h29m59.638235693s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection
Dec 28 15:58:59.767172856 (552h29m59.638255901s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767335827 (552h29m59.638418862s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767345114 (552h29m59.638428159s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767494889 (552h29m59.638577934s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767505590 (552h29m59.638588625s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767542280 (552h29m59.638625325s) [seg_pos={0 0}, upload=484402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:58:59.767650426 (552h29m59.638733481s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:58:59.767658240 (552h29m59.638741275s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:59:00.056943649 (552h29m59.928026694s) [upload=484402]: Done uploading segments...
Dec 28 15:59:00.056965510 (552h29m59.928048555s) [upload=484402]: Upload finished. err: <nil>
Duration: 42.362980278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
13.26055887s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:08:14 UTC)
orbiter warning: context deadline exceeded
Duration: 13.260586353s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
11.522191389s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:08:13 UTC)
orbiter warning: context deadline exceeded
Duration: 11.52235508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.X.XXX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X...] duration:
6.056243871s trace:
https://jaeger.dev.storj.io/trace/19fdb56c6a50b033
FAIL (2025-12-28 16:08:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ) context deadline exceeded
======LOGS======
Dec 28 16:04:37.528485291 (552h35m37.399568326s) [upload=484480]: Starting upload
Dec 28 16:04:37.528577968 (552h35m37.399661013s) [upload=484480]: Done starting upload
Dec 28 16:04:37.528640837 (552h35m37.399723882s) [upload=484480]: Uploading segments...
Dec 28 16:04:37.528978690 (552h35m37.400061735s) [upload=484480]: Got next segment. Inline: false
Dec 28 16:04:37.529013637 (552h35m37.400096682s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:04:37.529019909 (552h35m37.400102964s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:04:37.770378807 (552h35m37.641461842s) [seg_pos={0 0}, upload=484480]: Begin upload segment...
Dec 28 16:04:37.770445314 (552h35m37.641528359s) [seg_pos={0 0}, upload=484480]: Done begin upload segment.
Dec 28 16:04:37.770462446 (552h35m37.641545481s) [noise=true , address=145.79.227.47:4267 , piece_id=UPD34XVZABWACXLWURSGV7ANC5A7H426HE6WRHIYA6ARIYM3K36A, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.770542650 (552h35m37.641625685s) [noise=true , address=145.79.227.31:4798 , piece_id=ZA7N5IBZGUK47NTZFPPJFPY4IPWELZ6GUQ7T653OKI3LTFQ7XDVA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.771212455 (552h35m37.64229549s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.771731523 (552h35m37.642814558s) [noise=true , address=145.79.227.13:4804 , piece_id=GLEEPRVNORAMF6SK6RAMFCCS5NK5ROL3CDEJQVL4VJIW45WYOAHQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.772707151 (552h35m37.643790186s) [noise=true , address=145.79.227.4:4187 , piece_id=NSCA3K7MANN77LRJHXLBXIQGA7NEOR2Z2DYT7YK74KM33JJHWMGQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.772783767 (552h35m37.643866822s) [noise=true , address=77.90.31.13:4629 , piece_id=DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.773174921 (552h35m37.644257956s) [noise=true , address=145.79.227.7:4295 , piece_id=JGDXWWW7AGQYVF5ZL4ZXSCCFDNIEAYBGUOOKJVCQ3YXYA4J2PPSQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.773858233 (552h35m37.644941278s) [noise=true , address=145.79.227.16:4912 , piece_id=TJIAJ26SUAVZXCH23MYH3LCVKIZWU4F74LGF4CPMM5BW5MQOXQ5Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.774456882 (552h35m37.645539928s) [noise=true , address=145.79.227.41:4086 , piece_id=LO2ZSGF7Q2ZT5XAW6FK2Y2OX3V6JLJHETO7FXXMVFCSYFPOPUOMQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.774935053 (552h35m37.646018098s) [noise=true , address=145.79.227.19:4079 , piece_id=OO7QFBONFN2NUHA3ZXDWNXMVJ6X43FPTXSGAIIWHU5Q3DPXLBCCA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.775395209 (552h35m37.646478254s) [noise=true , address=77.90.31.11:4554 , piece_id=GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.775479529 (552h35m37.646562574s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.776129157 (552h35m37.647212232s) [noise=true , address=145.79.227.32:4835 , piece_id=CRBENMVGVYOOPGPXB6CIDJNFWH74LYCC4NZ4IJL5WKJMNRNIBDFA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.776762543 (552h35m37.647845578s) [noise=true , address=145.79.227.3:4150 , piece_id=LSXOVSZ2ATLO33UJZ74GSX6NACVIJE5SBF3XG4DOHTDHKLZTNJRA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.777421568 (552h35m37.648504623s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.777743390 (552h35m37.648826425s) [noise=true , address=145.79.227.27:4691 , piece_id=SRE4NU7DQVYPXB4AGW6WNYC6PGRMKPGKQBY7I7UUMT3IQEVPQ2UQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:39.482070524 (552h35m39.353153559s) [upload=484480]: Next returned nil segment
Dec 28 16:04:39.482082457 (552h35m39.353165492s) [upload=484480]: Waiting for error group managing segments...
Dec 28 16:04:39.766675369 (552h35m39.637758404s) [noise=true , address=145.79.227.31:4798 , piece_id=ZA7N5IBZGUK47NTZFPPJFPY4IPWELZ6GUQ7T653OKI3LTFQ7XDVA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.783697843 (552h35m39.654780888s) [noise=true , address=145.79.227.16:4912 , piece_id=TJIAJ26SUAVZXCH23MYH3LCVKIZWU4F74LGF4CPMM5BW5MQOXQ5Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.790128158 (552h35m39.661211193s) [noise=true , address=145.79.227.27:4691 , piece_id=SRE4NU7DQVYPXB4AGW6WNYC6PGRMKPGKQBY7I7UUMT3IQEVPQ2UQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.791084600 (552h35m39.662167635s) [noise=true , address=145.79.227.47:4267 , piece_id=UPD34XVZABWACXLWURSGV7ANC5A7H426HE6WRHIYA6ARIYM3K36A, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.791916333 (552h35m39.662999379s) [noise=true , address=145.79.227.41:4086 , piece_id=LO2ZSGF7Q2ZT5XAW6FK2Y2OX3V6JLJHETO7FXXMVFCSYFPOPUOMQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.792215172 (552h35m39.663298207s) [noise=true , address=145.79.227.19:4079 , piece_id=OO7QFBONFN2NUHA3ZXDWNXMVJ6X43FPTXSGAIIWHU5Q3DPXLBCCA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.797526476 (552h35m39.668609521s) [noise=true , address=145.79.227.32:4835 , piece_id=CRBENMVGVYOOPGPXB6CIDJNFWH74LYCC4NZ4IJL5WKJMNRNIBDFA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.799022304 (552h35m39.670105349s) [noise=true , address=145.79.227.3:4150 , piece_id=LSXOVSZ2ATLO33UJZ74GSX6NACVIJE5SBF3XG4DOHTDHKLZTNJRA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.801242962 (552h35m39.672326007s) [noise=true , address=145.79.227.7:4295 , piece_id=JGDXWWW7AGQYVF5ZL4ZXSCCFDNIEAYBGUOOKJVCQ3YXYA4J2PPSQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.802436274 (552h35m39.673519319s) [noise=true , address=145.79.227.4:4187 , piece_id=NSCA3K7MANN77LRJHXLBXIQGA7NEOR2Z2DYT7YK74KM33JJHWMGQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.807835044 (552h35m39.678918099s) [noise=true , address=145.79.227.13:4804 , piece_id=GLEEPRVNORAMF6SK6RAMFCCS5NK5ROL3CDEJQVL4VJIW45WYOAHQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.951953701 (552h35m39.823036736s) [noise=true , address=77.90.31.13:4629 , piece_id=DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.969307625 (552h35m39.84039067s) [noise=true , address=77.90.31.11:4554 , piece_id=GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.969326061 (552h35m39.840409096s) [seg_pos={0 0}, upload=484480]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:04:39.969478250 (552h35m39.840561285s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969486115 (552h35m39.84056916s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969707167 (552h35m39.840790202s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969713068 (552h35m39.840796103s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969853796 (552h35m39.840936831s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969859626 (552h35m39.840942662s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969874816 (552h35m39.840957861s) [seg_pos={0 0}, upload=484480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:04:39.969938877 (552h35m39.841021912s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:04:39.969943717 (552h35m39.841026752s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:04:40.259688480 (552h35m40.130771525s) [upload=484480]: Done uploading segments...
Dec 28 16:04:40.259727294 (552h35m40.130810329s) [upload=484480]: Upload finished. err: <nil>
Duration: 6.056273197s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XX.XXX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX.] duration:
3.727943548s trace:
https://jaeger.dev.storj.io/trace/21c3b5f5b9af47d4
FAIL (2025-12-28 16:08:02 UTC)
orbiter warning: context deadline exceeded
Duration: 3.727973356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
16.088441189s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:07:22 UTC)
orbiter warning: context deadline exceeded
Duration: 16.088467058s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
12.538848713s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 16:07:19 UTC)
orbiter warning: context deadline exceeded
Duration: 12.53887866s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX] duration:
5.001008005s trace:
https://jaeger.dev.storj.io/trace/21c3b5f5b9af47d4
FAIL (2025-12-28 16:07:11 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001036809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [XX.X.XXX.XX.XX...X.X........X.X.X.X....XXX...X...X.......X..] duration:
5.857895673s trace:
https://jaeger.dev.storj.io/trace/19fdb56c6a50b033
FAIL (2025-12-28 16:06:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ) context deadline exceeded
======LOGS======
Dec 28 16:04:37.528485291 (552h35m37.399568326s) [upload=484480]: Starting upload
Dec 28 16:04:37.528577968 (552h35m37.399661013s) [upload=484480]: Done starting upload
Dec 28 16:04:37.528640837 (552h35m37.399723882s) [upload=484480]: Uploading segments...
Dec 28 16:04:37.528978690 (552h35m37.400061735s) [upload=484480]: Got next segment. Inline: false
Dec 28 16:04:37.529013637 (552h35m37.400096682s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 16:04:37.529019909 (552h35m37.400102964s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 16:04:37.770378807 (552h35m37.641461842s) [seg_pos={0 0}, upload=484480]: Begin upload segment...
Dec 28 16:04:37.770445314 (552h35m37.641528359s) [seg_pos={0 0}, upload=484480]: Done begin upload segment.
Dec 28 16:04:37.770462446 (552h35m37.641545481s) [noise=true , address=145.79.227.47:4267 , piece_id=UPD34XVZABWACXLWURSGV7ANC5A7H426HE6WRHIYA6ARIYM3K36A, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.770542650 (552h35m37.641625685s) [noise=true , address=145.79.227.31:4798 , piece_id=ZA7N5IBZGUK47NTZFPPJFPY4IPWELZ6GUQ7T653OKI3LTFQ7XDVA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.771212455 (552h35m37.64229549s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.771731523 (552h35m37.642814558s) [noise=true , address=145.79.227.13:4804 , piece_id=GLEEPRVNORAMF6SK6RAMFCCS5NK5ROL3CDEJQVL4VJIW45WYOAHQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.772707151 (552h35m37.643790186s) [noise=true , address=145.79.227.4:4187 , piece_id=NSCA3K7MANN77LRJHXLBXIQGA7NEOR2Z2DYT7YK74KM33JJHWMGQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.772783767 (552h35m37.643866822s) [noise=true , address=77.90.31.13:4629 , piece_id=DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.773174921 (552h35m37.644257956s) [noise=true , address=145.79.227.7:4295 , piece_id=JGDXWWW7AGQYVF5ZL4ZXSCCFDNIEAYBGUOOKJVCQ3YXYA4J2PPSQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.773858233 (552h35m37.644941278s) [noise=true , address=145.79.227.16:4912 , piece_id=TJIAJ26SUAVZXCH23MYH3LCVKIZWU4F74LGF4CPMM5BW5MQOXQ5Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.774456882 (552h35m37.645539928s) [noise=true , address=145.79.227.41:4086 , piece_id=LO2ZSGF7Q2ZT5XAW6FK2Y2OX3V6JLJHETO7FXXMVFCSYFPOPUOMQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.774935053 (552h35m37.646018098s) [noise=true , address=145.79.227.19:4079 , piece_id=OO7QFBONFN2NUHA3ZXDWNXMVJ6X43FPTXSGAIIWHU5Q3DPXLBCCA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.775395209 (552h35m37.646478254s) [noise=true , address=77.90.31.11:4554 , piece_id=GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.775479529 (552h35m37.646562574s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.776129157 (552h35m37.647212232s) [noise=true , address=145.79.227.32:4835 , piece_id=CRBENMVGVYOOPGPXB6CIDJNFWH74LYCC4NZ4IJL5WKJMNRNIBDFA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.776762543 (552h35m37.647845578s) [noise=true , address=145.79.227.3:4150 , piece_id=LSXOVSZ2ATLO33UJZ74GSX6NACVIJE5SBF3XG4DOHTDHKLZTNJRA, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.777421568 (552h35m37.648504623s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:37.777743390 (552h35m37.648826425s) [noise=true , address=145.79.227.27:4691 , piece_id=SRE4NU7DQVYPXB4AGW6WNYC6PGRMKPGKQBY7I7UUMT3IQEVPQ2UQ, seg_pos={0 0}, upload=484480]: Uploading piece...
Dec 28 16:04:39.482070524 (552h35m39.353153559s) [upload=484480]: Next returned nil segment
Dec 28 16:04:39.482082457 (552h35m39.353165492s) [upload=484480]: Waiting for error group managing segments...
Dec 28 16:04:39.766675369 (552h35m39.637758404s) [noise=true , address=145.79.227.31:4798 , piece_id=ZA7N5IBZGUK47NTZFPPJFPY4IPWELZ6GUQ7T653OKI3LTFQ7XDVA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.783697843 (552h35m39.654780888s) [noise=true , address=145.79.227.16:4912 , piece_id=TJIAJ26SUAVZXCH23MYH3LCVKIZWU4F74LGF4CPMM5BW5MQOXQ5Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.790128158 (552h35m39.661211193s) [noise=true , address=145.79.227.27:4691 , piece_id=SRE4NU7DQVYPXB4AGW6WNYC6PGRMKPGKQBY7I7UUMT3IQEVPQ2UQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.791084600 (552h35m39.662167635s) [noise=true , address=145.79.227.47:4267 , piece_id=UPD34XVZABWACXLWURSGV7ANC5A7H426HE6WRHIYA6ARIYM3K36A, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.791916333 (552h35m39.662999379s) [noise=true , address=145.79.227.41:4086 , piece_id=LO2ZSGF7Q2ZT5XAW6FK2Y2OX3V6JLJHETO7FXXMVFCSYFPOPUOMQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.792215172 (552h35m39.663298207s) [noise=true , address=145.79.227.19:4079 , piece_id=OO7QFBONFN2NUHA3ZXDWNXMVJ6X43FPTXSGAIIWHU5Q3DPXLBCCA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.797526476 (552h35m39.668609521s) [noise=true , address=145.79.227.32:4835 , piece_id=CRBENMVGVYOOPGPXB6CIDJNFWH74LYCC4NZ4IJL5WKJMNRNIBDFA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.799022304 (552h35m39.670105349s) [noise=true , address=145.79.227.3:4150 , piece_id=LSXOVSZ2ATLO33UJZ74GSX6NACVIJE5SBF3XG4DOHTDHKLZTNJRA, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.801242962 (552h35m39.672326007s) [noise=true , address=145.79.227.7:4295 , piece_id=JGDXWWW7AGQYVF5ZL4ZXSCCFDNIEAYBGUOOKJVCQ3YXYA4J2PPSQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.802436274 (552h35m39.673519319s) [noise=true , address=145.79.227.4:4187 , piece_id=NSCA3K7MANN77LRJHXLBXIQGA7NEOR2Z2DYT7YK74KM33JJHWMGQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.807835044 (552h35m39.678918099s) [noise=true , address=145.79.227.13:4804 , piece_id=GLEEPRVNORAMF6SK6RAMFCCS5NK5ROL3CDEJQVL4VJIW45WYOAHQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.951953701 (552h35m39.823036736s) [noise=true , address=77.90.31.13:4629 , piece_id=DDW3M67Y3H7IGIDTOXY3LR7IRU6ALMLFULRVALUMYXC327C3BD5Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.969307625 (552h35m39.84039067s) [noise=true , address=77.90.31.11:4554 , piece_id=GOIJLZJD7EIHKPBV4D5LWSU63PXW27RFSN7ACIAIK47NL6JWBITQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: <nil>
Dec 28 16:04:39.969326061 (552h35m39.840409096s) [seg_pos={0 0}, upload=484480]: Segment reached optimal threshold of 13 pieces.
Dec 28 16:04:39.969478250 (552h35m39.840561285s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969486115 (552h35m39.84056916s) [noise=true , address=145.79.227.49:4338 , piece_id=24SI7BCBYGBAXXBOHQXGYBS4LVKPYSYVXUIJ3TZXXVTYQIRLDE4Q, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969707167 (552h35m39.840790202s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969713068 (552h35m39.840796103s) [noise=true , address=45.91.116.9:4261 , piece_id=ZDPQIDPHDFA5EVYP4OUZCDH5YFJLQZBWCEJJ3EFTWZL54CZMRT3A, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969853796 (552h35m39.840936831s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 28 16:04:39.969859626 (552h35m39.840942662s) [noise=true , address=145.79.227.35:4481 , piece_id=IJN2LIE5EXAASF6YVGOUBHF3KGEIFH2EWYD4V2XBL7HKTA4FRUAQ, seg_pos={0 0}, upload=484480]: Long tail cancellation due to optimal threshold
Dec 28 16:04:39.969874816 (552h35m39.840957861s) [seg_pos={0 0}, upload=484480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 16:04:39.969938877 (552h35m39.841021912s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 16:04:39.969943717 (552h35m39.841026752s) [upload=484480]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 16:04:40.259688480 (552h35m40.130771525s) [upload=484480]: Done uploading segments...
Dec 28 16:04:40.259727294 (552h35m40.130810329s) [upload=484480]: Upload finished. err: <nil>
Duration: 5.857931612s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.695595359s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:06:50 UTC)
orbiter warning: context deadline exceeded
Duration: 5.695628562s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X........................X.....................X.........] duration:
15.60629862s trace:
https://jaeger.dev.storj.io/trace/556a3fb5861f29f
FAIL (2025-12-28 16:06:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
e=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574577548 (552h29m56.445660593s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.574886507 (552h29m56.445969542s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575168363 (552h29m56.446251398s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575547866 (552h29m56.446630911s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.575844602 (552h29m56.446927647s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576175541 (552h29m56.447258576s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.576516070 (552h29m56.447599105s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.578534654 (552h29m56.449617699s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.580192551 (552h29m56.451275596s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.581999100 (552h29m56.453082145s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.584764987 (552h29m56.455848022s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.586884734 (552h29m56.457967799s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.588442480 (552h29m56.459525525s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.590653229 (552h29m56.461736265s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:56.592369818 (552h29m56.463452853s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Uploading piece...
Dec 28 15:58:57.859277503 (552h29m57.730360538s) [upload=484402]: Next returned nil segment
Dec 28 15:58:57.859315846 (552h29m57.730398891s) [upload=484402]: Waiting for error group managing segments...
Dec 28 15:58:58.228560330 (552h29m58.099643365s) [noise=true , address=64.49.13.26:4954 , piece_id=M2XTTVANSJRWA3ZOKWW6TY75J7SH6ZAY64BS4BP7SHWLGVZQSNMQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.252716918 (552h29m58.123799953s) [noise=true , address=64.49.13.170:4728 , piece_id=GLLYT75A6HPLO7HJSHLFIEDACKFZUMJORIIU73SA7UFOGOHW3TTA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.284245663 (552h29m58.155328698s) [noise=true , address=64.49.13.140:4516 , piece_id=7X6B7K25VYCURW3FLIZCXEI7PXYETFNKDK4LX553M5RLUZKH3YHQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.517657659 (552h29m58.388740694s) [noise=true , address=64.49.13.137:4501 , piece_id=MOSBMHVIRHGDLOF6UHCEQJLPUHOPKNOT67X5U4EUG2NDQDYAI65Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.677500887 (552h29m58.548583922s) [noise=true , address=45.91.116.8:4055 , piece_id=K6ERECS6KI2VIHYGY6GSQ4HFBLDU4GOZ7SN4DTPHNTMGZHUS6E6A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.712891747 (552h29m58.583974792s) [noise=true , address=77.90.31.14:4686 , piece_id=V42OZVXLGURDB4QRSLVKGTAL3WEAME4BZFGS3JUMG2M7P5WSN77A, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:58.885850299 (552h29m58.756933344s) [noise=true , address=77.90.31.18:4812 , piece_id=SBVY44WVNJTZ4VHOXTLUDCKVST5NP5UCTDYN5UCIVLINMTIIJNCQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.312640901 (552h29m59.183723946s) [noise=true , address=77.90.31.13:4639 , piece_id=IROPCGBCD7AQTKF7XOYR552XEVL366AB2PIMZGDHXF7KQQOEQ6SA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.375303962 (552h29m59.246386997s) [noise=true , address=145.79.227.44:4203 , piece_id=OURPQ2QCNM6WC643FZN2GZUU7IPS7TMB2ERIDW7CKHW4JG4CQYAA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.424009806 (552h29m59.295092851s) [noise=true , address=145.79.227.40:4071 , piece_id=RWGYCHC5XSDBIGY3JQ6CU6CVKDDQYQH2GSF2U7TO26E2NRLFQDBQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.442462813 (552h29m59.313545858s) [noise=true , address=145.79.227.33:4878 , piece_id=DICRMWSI7H4EGZH67QUI3ROLJ76ANVGWT6EKSQGXXECDSRRLPGSQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.446628285 (552h29m59.31771132s) [noise=true , address=145.79.227.30:4775 , piece_id=76C6HT3XEO42X4YV4NDTXEAIUVGNYYJWIUC7TQQFU3WVIZKFTLZQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.568135996 (552h29m59.439219031s) [noise=true , address=145.79.227.34:4910 , piece_id=FQAH4XZDWX5SE2YCAORAJBI5IQMBEB25XBDRM3NY3RQNYHLWCFWQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766737718 (552h29m59.637820763s) [noise=true , address=145.79.227.43:4187 , piece_id=JZQ7CX2XK7QZ7WUJ3P2DGB6KYXT5MSQLP62LDSW6BP3WVARANBIQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: <nil>
Dec 28 15:58:59.766764528 (552h29m59.637847563s) [seg_pos={0 0}, upload=484402]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:58:59.767019034 (552h29m59.638102069s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767029423 (552h29m59.638112458s) [noise=true , address=45.91.116.12:4229 , piece_id=3EAJTXD2V5DG6AD6S5X4TXBSN6D3476ONISUAH2KO7JH2PCYSWRQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767152658 (552h29m59.638235693s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40894->64.49.13.169:4710: use of closed network connection
Dec 28 15:58:59.767172856 (552h29m59.638255901s) [noise=true , address=64.49.13.169:4710 , piece_id=PHX3JDQQEVJWHRJNLYF2JV7NUSRSFQIAGDF3VUI42CN6JMMQDOFA, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767335827 (552h29m59.638418862s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767345114 (552h29m59.638428159s) [noise=true , address=193.149.164.143:4695 , piece_id=WYGBI7NYYQZX2GREZXQQABIU4ZQOZNIZQIB452GGO6BC5U7CZJXQ, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767494889 (552h29m59.638577934s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: expected piece hash; context canceled; EOF
Dec 28 15:58:59.767505590 (552h29m59.638588625s) [noise=true , address=193.149.164.143:4702 , piece_id=CTUGCVON2K7SQH7QFCONBHL7SYK3KAGS2FAD5M7ZI4RKRSTEVZ5Q, seg_pos={0 0}, upload=484402]: Long tail cancellation due to optimal threshold
Dec 28 15:58:59.767542280 (552h29m59.638625325s) [seg_pos={0 0}, upload=484402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:58:59.767650426 (552h29m59.638733481s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:58:59.767658240 (552h29m59.638741275s) [upload=484402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:59:00.056943649 (552h29m59.928026694s) [upload=484402]: Done uploading segments...
Dec 28 15:59:00.056965510 (552h29m59.928048555s) [upload=484402]: Upload finished. err: <nil>
Duration: 15.606346251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [.XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXX] duration:
5.000799599s trace:
https://jaeger.dev.storj.io/trace/4d0a6d294ae5fead
FAIL (2025-12-28 16:06:35 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000825297s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
16.209123631s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 16:06:33 UTC)
orbiter warning: context deadline exceeded
Duration: 16.20914992s
core
uplink
us-select-1
us1-placement-20
|