Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X...X..X...................] duration:
5.366264848s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:25:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.366291938s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration:
5.578066435s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:25:45 UTC)
orbiter warning: context deadline exceeded
Duration: 5.57809588s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......X] duration:
30.004747632s trace:
https://jaeger.dev.storj.io/trace/1dc4879c98bae820
FAIL (2025-12-30 17:25:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: LODK3KFRRMWPWZ4SAOF66D7DG6HQWWCOQCSGD2J6RXZ5OCYF543A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 5T3JCBIKPGYX7BZSXMKURT6AKL2FKRJ3E6FWJHC2ZKNNKMMJ7TTA) context deadline exceeded
======LOGS======
Dec 30 17:25:03.297991827 (601h56m3.169074872s) [upload=527577]: Starting upload
Dec 30 17:25:03.298092489 (601h56m3.169175564s) [upload=527577]: Done starting upload
Dec 30 17:25:03.298233337 (601h56m3.169316392s) [upload=527577]: Uploading segments...
Dec 30 17:25:03.298343226 (601h56m3.169426292s) [upload=527577]: Got next segment. Inline: false
Dec 30 17:25:03.298411937 (601h56m3.169494982s) [upload=527577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:25:03.298419622 (601h56m3.169502667s) [upload=527577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:25:03.552214259 (601h56m3.423297294s) [seg_pos={0 0}, upload=527577]: Begin upload segment...
Dec 30 17:25:03.552278982 (601h56m3.423362027s) [seg_pos={0 0}, upload=527577]: Done begin upload segment.
Dec 30 17:25:03.552319549 (601h56m3.423402594s) [noise=true , address=145.79.227.21:4473 , piece_id=QO3FBXOBMWFVMIGPW3BZESSEHHZ5OVWQWH5SCRLWR4QIUOYXG6SQ, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.552425731 (601h56m3.423508807s) [noise=true , address=45.91.116.10:4082 , piece_id=W3UDZ7N2KKUHORTU6MXT2SCTMIBKSBAYOWIQ4A6TXEYPX4IOK76Q, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.553282984 (601h56m3.424366039s) [noise=true , address=145.79.227.12:4765 , piece_id=6SMCVQRELJQDKMTGV3R3IN62NXQZKSIBT67E7YKMK5ZVDKBARZEA, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.553723824 (601h56m3.424806869s) [noise=true , address=145.79.227.33:4871 , piece_id=NSMXLMYUJK3RSMTRPZ7A3ZIFLEIENKVKBKZXXC5Q2JNR75HRAVKA, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.553755384 (601h56m3.424838429s) [noise=true , address=77.90.31.10:4519 , piece_id=IGHWRT7SPFDSDIQZPPENW7CHMYBTH5UZZ7JLYNP6Z7O3HYCAXUZQ, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.554267959 (601h56m3.425351004s) [noise=true , address=145.79.227.10:4402 , piece_id=WZPZ5HPOJINWLBNPUQKRF4G3LYVRBM57X44EK3QDRNLQRSN2BY6A, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.554432733 (601h56m3.425515779s) [noise=true , address=77.90.31.19:4842 , piece_id=LODK3KFRRMWPWZ4SAOF66D7DG6HQWWCOQCSGD2J6RXZ5OCYF543A, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.554745760 (601h56m3.425828805s) [noise=true , address=45.91.116.7:4010 , piece_id=V7STIO43GE4ICE74L7VYS6EMBMXX6KWKDZOE3F7SYBLNFJA3UCJA, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.555090416 (601h56m3.426173461s) [noise=true , address=145.79.227.7:4293 , piece_id=FY7BGB53X5BQTQFMVWURNK32CTA5U2DLG44ECJLFXFSB7ED5VTZQ, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.555164066 (601h56m3.426247111s) [noise=true , address=45.91.116.13:4155 , piece_id=5T3JCBIKPGYX7BZSXMKURT6AKL2FKRJ3E6FWJHC2ZKNNKMMJ7TTA, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.555629713 (601h56m3.426712758s) [noise=true , address=45.91.116.11:4118 , piece_id=7SNKMAI5T5ZCKUVM57RFUMLYQBCYO5YGMIHZ7FBUCKJM7BJVHFBQ, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.555761363 (601h56m3.426844408s) [noise=true , address=145.79.227.14:4838 , piece_id=AOXTSNDXQBGNTBMHKLAAM6R2NS4UBFKT5IY6ULWS2OPHZ7RRFL2A, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.556042378 (601h56m3.427125423s) [noise=true , address=145.79.227.13:4803 , piece_id=7CPJ6QC3QEEKCO6AGRQL3LNRN6YZVNKC6GTMNSPHHOGZRQ72OHYA, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.556449694 (601h56m3.427532739s) [noise=true , address=145.79.227.9:4366 , piece_id=6YBSAYOGQSTGCSLBKF4K7RNZLZGEXZZHOGY5HGSZUTWC7DADQVNA, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.566473902 (601h56m3.437556947s) [noise=true , address=145.79.227.23:4546 , piece_id=W5CFC4ZCUYZGDTZCKEASKLC4OKQOVCXCZHDISWBJYAC425HRGKHQ, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:03.569188441 (601h56m3.440271496s) [noise=true , address=145.79.227.2:4113 , piece_id=DXDDACHHY7M4QLXGG5KLWI7ODBX4MQLRXB4K54QEEFX3NMQSLBJQ, seg_pos={0 0}, upload=527577]: Uploading piece...
Dec 30 17:25:05.162191823 (601h56m5.033274878s) [upload=527577]: Next returned nil segment
Dec 30 17:25:05.162240596 (601h56m5.033323641s) [upload=527577]: Waiting for error group managing segments...
Dec 30 17:25:05.547350265 (601h56m5.4184333s) [noise=true , address=145.79.227.9:4366 , piece_id=6YBSAYOGQSTGCSLBKF4K7RNZLZGEXZZHOGY5HGSZUTWC7DADQVNA, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.548487209 (601h56m5.419570254s) [noise=true , address=145.79.227.12:4765 , piece_id=6SMCVQRELJQDKMTGV3R3IN62NXQZKSIBT67E7YKMK5ZVDKBARZEA, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.550298468 (601h56m5.421381513s) [noise=true , address=145.79.227.13:4803 , piece_id=7CPJ6QC3QEEKCO6AGRQL3LNRN6YZVNKC6GTMNSPHHOGZRQ72OHYA, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.556481872 (601h56m5.427564907s) [noise=true , address=145.79.227.7:4293 , piece_id=FY7BGB53X5BQTQFMVWURNK32CTA5U2DLG44ECJLFXFSB7ED5VTZQ, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.556826308 (601h56m5.427909343s) [noise=true , address=145.79.227.23:4546 , piece_id=W5CFC4ZCUYZGDTZCKEASKLC4OKQOVCXCZHDISWBJYAC425HRGKHQ, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.559044471 (601h56m5.430127516s) [noise=true , address=145.79.227.10:4402 , piece_id=WZPZ5HPOJINWLBNPUQKRF4G3LYVRBM57X44EK3QDRNLQRSN2BY6A, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.571668209 (601h56m5.442751254s) [noise=true , address=145.79.227.14:4838 , piece_id=AOXTSNDXQBGNTBMHKLAAM6R2NS4UBFKT5IY6ULWS2OPHZ7RRFL2A, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.576537461 (601h56m5.447620496s) [noise=true , address=145.79.227.33:4871 , piece_id=NSMXLMYUJK3RSMTRPZ7A3ZIFLEIENKVKBKZXXC5Q2JNR75HRAVKA, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.596605824 (601h56m5.467688859s) [noise=true , address=145.79.227.2:4113 , piece_id=DXDDACHHY7M4QLXGG5KLWI7ODBX4MQLRXB4K54QEEFX3NMQSLBJQ, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.622582736 (601h56m5.493665781s) [noise=true , address=145.79.227.21:4473 , piece_id=QO3FBXOBMWFVMIGPW3BZESSEHHZ5OVWQWH5SCRLWR4QIUOYXG6SQ, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.729253521 (601h56m5.600336556s) [noise=true , address=77.90.31.19:4842 , piece_id=LODK3KFRRMWPWZ4SAOF66D7DG6HQWWCOQCSGD2J6RXZ5OCYF543A, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.851707139 (601h56m5.722790184s) [noise=true , address=77.90.31.10:4519 , piece_id=IGHWRT7SPFDSDIQZPPENW7CHMYBTH5UZZ7JLYNP6Z7O3HYCAXUZQ, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.919200545 (601h56m5.79028358s) [noise=true , address=45.91.116.13:4155 , piece_id=5T3JCBIKPGYX7BZSXMKURT6AKL2FKRJ3E6FWJHC2ZKNNKMMJ7TTA, seg_pos={0 0}, upload=527577]: Done uploading piece. err: <nil>
Dec 30 17:25:05.919222156 (601h56m5.790305201s) [seg_pos={0 0}, upload=527577]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:25:05.919521627 (601h56m5.790604672s) [noise=true , address=45.91.116.11:4118 , piece_id=7SNKMAI5T5ZCKUVM57RFUMLYQBCYO5YGMIHZ7FBUCKJM7BJVHFBQ, seg_pos={0 0}, upload=527577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 30 17:25:05.919530584 (601h56m5.790613619s) [noise=true , address=45.91.116.11:4118 , piece_id=7SNKMAI5T5ZCKUVM57RFUMLYQBCYO5YGMIHZ7FBUCKJM7BJVHFBQ, seg_pos={0 0}, upload=527577]: Long tail cancellation due to optimal threshold
Dec 30 17:25:05.919681551 (601h56m5.790764586s) [noise=true , address=45.91.116.7:4010 , piece_id=V7STIO43GE4ICE74L7VYS6EMBMXX6KWKDZOE3F7SYBLNFJA3UCJA, seg_pos={0 0}, upload=527577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 30 17:25:05.919688133 (601h56m5.790771178s) [noise=true , address=45.91.116.7:4010 , piece_id=V7STIO43GE4ICE74L7VYS6EMBMXX6KWKDZOE3F7SYBLNFJA3UCJA, seg_pos={0 0}, upload=527577]: Long tail cancellation due to optimal threshold
Dec 30 17:25:05.919946626 (601h56m5.791029671s) [noise=true , address=45.91.116.10:4082 , piece_id=W3UDZ7N2KKUHORTU6MXT2SCTMIBKSBAYOWIQ4A6TXEYPX4IOK76Q, seg_pos={0 0}, upload=527577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:37088->45.91.116.10:4082: use of closed network connection; context canceled
Dec 30 17:25:05.919953900 (601h56m5.791036935s) [noise=true , address=45.91.116.10:4082 , piece_id=W3UDZ7N2KKUHORTU6MXT2SCTMIBKSBAYOWIQ4A6TXEYPX4IOK76Q, seg_pos={0 0}, upload=527577]: Long tail cancellation due to optimal threshold
Dec 30 17:25:05.919970170 (601h56m5.791053205s) [seg_pos={0 0}, upload=527577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:25:05.920042940 (601h56m5.791125975s) [upload=527577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:25:05.920048189 (601h56m5.791131234s) [upload=527577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:25:06.206855869 (601h56m6.077938914s) [upload=527577]: Done uploading segments...
Dec 30 17:25:06.206876127 (601h56m6.077959162s) [upload=527577]: Upload finished. err: <nil>
Duration: 30.004864545s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [...XX......X.XX................X..X....X.XX..X..X......X....] duration:
3.421239361s trace:
https://jaeger.dev.storj.io/trace/7167bc8a3e953d5a
FAIL (2025-12-30 17:25:25 UTC)
orbiter warning: context deadline exceeded
Duration: 3.421266242s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X...X..X..................] duration:
6.539861645s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:25:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.539888035s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration:
5.977829695s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:25:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.977859251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......] duration:
5.978539876s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:24:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 5.978565805s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X.......X................................X...............] duration:
36.569688267s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:24:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 36.56970603s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [X...XX......X.XX................X..X....X.XX..X..X......X...] duration:
3.710353898s trace:
https://jaeger.dev.storj.io/trace/7167bc8a3e953d5a
FAIL (2025-12-30 17:24:33 UTC)
orbiter warning: context deadline exceeded
Duration: 3.710378445s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................X...X........................] duration:
5.584079143s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:24:22 UTC)
orbiter warning: context deadline exceeded
Duration: 5.584105875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X.......X................................X..............] duration:
33.99666978s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:24:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 33.996697381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................X...X..X.................] duration:
6.216125927s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:24:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.216153468s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX.....] duration:
6.069793388s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:23:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 6.069848473s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X...X.......................] duration:
5.806289873s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:23:58 UTC)
orbiter warning: context deadline exceeded
Duration: 5.80631478s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................X...X..X................] duration:
9.610283811s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:23:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.610309049s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [.X...XX......X.XX................X..X....X.XX..X..X......X..] duration:
3.464779286s trace:
https://jaeger.dev.storj.io/trace/7167bc8a3e953d5a
FAIL (2025-12-30 17:23:39 UTC)
orbiter warning: context deadline exceeded
Duration: 3.464802701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX....] duration:
6.289968198s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:23:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 6.289990241s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................X...X......................] duration:
5.603237205s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:23:16 UTC)
orbiter warning: context deadline exceeded
Duration: 5.603342735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X...X..X...............] duration:
6.068481458s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:23:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.068557193s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX...] duration:
6.252895336s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:23:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 6.252929132s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X.......X................................X.............] duration:
35.934897722s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:22:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 35.93492734s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [X.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX..] duration:
6.224058445s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:22:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 6.224086869s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [..X...XX......X.XX................X..X....X.XX..X..X......X.] duration:
3.676227226s trace:
https://jaeger.dev.storj.io/trace/7167bc8a3e953d5a
FAIL (2025-12-30 17:22:48 UTC)
orbiter warning: context deadline exceeded
Duration: 3.676251993s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................X...X..X..............] duration:
6.97909179s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:22:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.979113482s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration:
5.97941637s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:22:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.979440385s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [X..X...XX......X.XX................X..X....X.XX..X..X......X] duration:
5.000985315s trace:
https://jaeger.dev.storj.io/trace/7167bc8a3e953d5a
FAIL (2025-12-30 17:22:25 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001007627s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X...X..X.............] duration:
6.293590251s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:22:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.293614928s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X.......X................................X............] duration:
32.476089096s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:22:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 32.476107701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [XX..X...XX......X.XX................X..X....X.XX..X..X......] duration:
3.598351886s trace:
https://jaeger.dev.storj.io/trace/65d425bd21adcb47
FAIL (2025-12-30 17:22:00 UTC)
orbiter warning: context deadline exceeded
Duration: 3.598379138s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX.] duration:
5.872482394s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:21:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 5.872516269s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.......X................................X...........] duration:
49.198706571s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:21:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 49.198728735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................X...X....................] duration:
6.248410307s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:21:36 UTC)
orbiter warning: context deadline exceeded
Duration: 6.248472326s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................X...X..X............] duration:
7.095505095s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:21:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.095536524s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [.XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX] duration:
30.003327622s trace:
https://jaeger.dev.storj.io/trace/3d42995ae2b0b3f3
FAIL (2025-12-30 17:21:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ) context deadline exceeded
======LOGS======
Dec 30 17:20:59.800058349 (601h51m59.671141394s) [upload=527514]: Starting upload
Dec 30 17:20:59.800114176 (601h51m59.671197211s) [upload=527514]: Done starting upload
Dec 30 17:20:59.800348051 (601h51m59.671431096s) [upload=527514]: Uploading segments...
Dec 30 17:20:59.800399419 (601h51m59.671482454s) [upload=527514]: Got next segment. Inline: false
Dec 30 17:20:59.800428124 (601h51m59.671511159s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:59.800432582 (601h51m59.671515617s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:21:00.049697785 (601h51m59.92078083s) [seg_pos={0 0}, upload=527514]: Begin upload segment...
Dec 30 17:21:00.049762208 (601h51m59.920845243s) [seg_pos={0 0}, upload=527514]: Done begin upload segment.
Dec 30 17:21:00.049787506 (601h51m59.920870541s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049864042 (601h51m59.920947087s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.049937051 (601h51m59.921020096s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050381097 (601h51m59.921464142s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050397728 (601h51m59.921480763s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050783472 (601h51m59.921866507s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.050794734 (601h51m59.921877769s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051181240 (601h51m59.922264285s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.051672997 (601h51m59.922756042s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052064322 (601h51m59.923147357s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052482738 (601h51m59.923565783s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.052911504 (601h51m59.923994539s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053348747 (601h51m59.924431793s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.053847707 (601h51m59.924930752s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.054271694 (601h51m59.9253549s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:00.055695856 (601h51m59.926778901s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Uploading piece...
Dec 30 17:21:01.710194595 (601h52m1.58127764s) [upload=527514]: Next returned nil segment
Dec 30 17:21:01.710220193 (601h52m1.581303238s) [upload=527514]: Waiting for error group managing segments...
Dec 30 17:21:02.038242480 (601h52m1.909325525s) [noise=true , address=145.79.227.10:4403 , piece_id=COIALEQY4XJCJG4MQTJWVUK6P4RKNEATAM4K76Z7A4EWOQKNIAGA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.038907467 (601h52m1.909990512s) [noise=true , address=145.79.227.4:4186 , piece_id=FWDUJCWVMSLWC53BMKUWF554YL3TWPJ6OP42TFVOI36VAGLP5NYA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.041080314 (601h52m1.912163349s) [noise=true , address=145.79.227.13:4804 , piece_id=L6JCX7PN4UFRU2NY7ANVZCHC5Q3766XUEIISB2YPIXCD4GRNMRXA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.048034645 (601h52m1.91911769s) [noise=true , address=145.79.227.39:4014 , piece_id=5XA3EJ2OTUCS7KQ35KEVT7T2ZKPCMMRZSPTEADRNADTFVWTVIM5A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.054448889 (601h52m1.925531944s) [noise=true , address=145.79.227.20:4437 , piece_id=ICRUCKSWGZ6PSFSS24FUDOF6ML6B7MRNVOYPDRDJLVY4XMHZBR3A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.061739461 (601h52m1.932822506s) [noise=true , address=145.79.227.50:4375 , piece_id=Y7UQEC4HCCOWAULVZABP2LKT3AQGZUMKM6HEH77OZPU7ZMDRMANQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.063368603 (601h52m1.934451638s) [noise=true , address=145.79.227.9:4367 , piece_id=XG6WMWH7F6WXNERVRH62Q7HAZTDPUFONSCUVTLOUO2KNQVVYEMZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.065656991 (601h52m1.936740046s) [noise=true , address=145.79.227.2:4113 , piece_id=BRGBIET2HD2VUV4YCDJJOUV2KAFLXURAIJ2RMSHA6P4OB7JUVC7Q, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.072104217 (601h52m1.943187262s) [noise=true , address=145.79.227.52:4446 , piece_id=RLCMDKZLZMR4M4NYDU3HXEXWXK6TSGJ5L4AWLY2LVGF5BIPXZ6EQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.131755521 (601h52m2.002838566s) [noise=true , address=145.79.227.23:4546 , piece_id=M3F54H4IM5STC6LVUPPU7X7KZCUBLFTZF2MJQ33RZN5U463WVW6A, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.150415261 (601h52m2.021498306s) [noise=true , address=145.79.227.38:4978 , piece_id=ZN7ONRUVPSMP7L2WKYSMRHXOQ3KGFSYY3UQSFTZNVZD3H7MTHXJQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.451295226 (601h52m2.322378272s) [noise=true , address=45.91.116.7:4010 , piece_id=BIYRNAXABMCB5Y6TOYX2QZ32SRGRIQ6CSJWPVPVZATJ6WH4WP3UQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473377183 (601h52m2.344460218s) [noise=true , address=45.91.116.12:4226 , piece_id=SZY7MP3FO4REFYXUGWNRNKMDRMQ4SGMSDN2NYGV6DUNK4RVI4AZQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: <nil>
Dec 30 17:21:02.473403453 (601h52m2.344486498s) [seg_pos={0 0}, upload=527514]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:21:02.473681052 (601h52m2.344764137s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.473690780 (601h52m2.344773825s) [noise=true , address=145.79.227.15:4876 , piece_id=KL4DC7KDXE4FEAEXSNE46EY3KRM3ELVPG3QVW6YH36SAPYY3WSUQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.473995911 (601h52m2.345078956s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:35472->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:21:02.474017241 (601h52m2.345100296s) [noise=true , address=77.90.31.10:4519 , piece_id=JF3ZZVAQOINJSDUH3JAWFQP22N3QVKFRY7GPV3USS6ADAYFXHBNA, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474028243 (601h52m2.345111288s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 30 17:21:02.474036418 (601h52m2.345119463s) [noise=true , address=45.91.116.9:4262 , piece_id=LRPA3G75I35S4LNOQ2GQO54LBSX54XMCASY2HFEYDKLKHKEDZJOQ, seg_pos={0 0}, upload=527514]: Long tail cancellation due to optimal threshold
Dec 30 17:21:02.474057759 (601h52m2.345140794s) [seg_pos={0 0}, upload=527514]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:21:02.474142370 (601h52m2.345225405s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:21:02.474147299 (601h52m2.345230334s) [upload=527514]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:21:02.774556405 (601h52m2.64563945s) [upload=527514]: Done uploading segments...
Dec 30 17:21:02.774587495 (601h52m2.64567054s) [upload=527514]: Upload finished. err: <nil>
Duration: 30.00335377s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................X...X...................] duration:
19.278044282s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:21:27 UTC)
orbiter warning: context deadline exceeded
Duration: 19.278068308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [XXX..X...XX......X.XX................X..X....X.XX..X..X.....] duration:
3.449693099s trace:
https://jaeger.dev.storj.io/trace/65d425bd21adcb47
FAIL (2025-12-30 17:21:02 UTC)
orbiter warning: context deadline exceeded
Duration: 3.449732244s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XX] duration:
30.003016848s trace:
https://jaeger.dev.storj.io/trace/2f92a0f5bb43f872
FAIL (2025-12-30 17:20:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: W3PMPHBXIF534CW6PYPKW4YUIMCRMX3GGNHBU57444KXGOXBQUXQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 5TWFFB5AK46UWU7GYO46FCMZ7ZNORAQB7Y2AO4MXGFUKMOMC77ZA) context deadline exceeded
======LOGS======
=527506]: Starting upload
Dec 30 17:20:26.609031732 (601h51m26.480114777s) [upload=527506]: Done starting upload
Dec 30 17:20:26.609096965 (601h51m26.48018001s) [upload=527506]: Uploading segments...
Dec 30 17:20:26.609397177 (601h51m26.480480212s) [upload=527506]: Got next segment. Inline: false
Dec 30 17:20:26.609428136 (601h51m26.480511171s) [upload=527506]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:20:26.609432935 (601h51m26.48051598s) [upload=527506]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:20:26.860216870 (601h51m26.731299905s) [seg_pos={0 0}, upload=527506]: Begin upload segment...
Dec 30 17:20:26.860262466 (601h51m26.731345501s) [seg_pos={0 0}, upload=527506]: Done begin upload segment.
Dec 30 17:20:26.860287855 (601h51m26.7313709s) [noise=true , address=45.91.116.14:4192 , piece_id=5TWFFB5AK46UWU7GYO46FCMZ7ZNORAQB7Y2AO4MXGFUKMOMC77ZA, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.860386502 (601h51m26.731469537s) [noise=true , address=77.90.31.19:4844 , piece_id=AKUW3G7O3C5UEEGHJMWRMWP72D5V3LJJPOVL5Y5QLL6KGDEALFZQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.860993448 (601h51m26.732076493s) [noise=true , address=77.90.31.11:4554 , piece_id=W3PMPHBXIF534CW6PYPKW4YUIMCRMX3GGNHBU57444KXGOXBQUXQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.861320641 (601h51m26.732403676s) [noise=true , address=77.90.31.17:4770 , piece_id=BX5UJPEJLJJ5AA57RQRYJHG6MTGL7W6GQ3XRSATWRTUIGE2M5FPQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.861621003 (601h51m26.732704038s) [noise=true , address=145.79.227.8:4331 , piece_id=3D56OR643CBBW6JNXSOA7NRXQBY32OVMLRDZAMHKP5C7OM67FZFQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.861990436 (601h51m26.733073481s) [noise=true , address=45.91.116.4:4939 , piece_id=3WU5VRJO2LG747WQQ7DRZE7NE2BVDJ7ARU2ESSUKWZAR5SQ5H46A, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.862274107 (601h51m26.733357142s) [noise=true , address=77.90.31.16:4735 , piece_id=RGECA3YAS6TBWBLWIMP2G4TCG7V36RVRQDFSYTBNHU3LUG5NF3FQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.862567175 (601h51m26.73365021s) [noise=true , address=45.91.116.5:4297 , piece_id=6WC3XS7ZDCWLIFBTT3IGBV5XKQNCPSLM2NY5FVE356UBAE6PRZRQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.862916580 (601h51m26.733999625s) [noise=true , address=145.79.227.16:4909 , piece_id=WXTUOKURDVSP7K7OGB6JJVV722W6LRF7QF6MKQMEVK3WCV5HQ7GQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.864332617 (601h51m26.735415652s) [noise=true , address=145.79.227.39:4013 , piece_id=NHVSOIDKDXWCGSXZZA3J2J4LSMJCQ4OQOLUZ6NT45BDTXHC56DDQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.864636985 (601h51m26.73572003s) [noise=true , address=145.79.227.46:4601 , piece_id=P2EAZYLYUBMYAW3WLZTDZTXKTW2PNBP7CSF54LAT22MP23XPK6YQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.866915144 (601h51m26.737998189s) [noise=true , address=145.79.227.51:4411 , piece_id=WEAPWVJYM3WYMGCMZCYRJWC4U3YPIPWR3D3AN3WBROHRCY35RJ6A, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.869019310 (601h51m26.740102345s) [noise=true , address=145.79.227.45:4229 , piece_id=HPRBYTHNO4OUS6N4CHJSKIGX3KIA5L2E65G4FFVEYJHNLEOVI3GA, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.872379780 (601h51m26.743462815s) [noise=true , address=77.90.31.14:4664 , piece_id=WHGXL4LIUJRKVET4WEM2G4467UM7X64FV5W5KBXU7DZCDARBSEUA, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.875704961 (601h51m26.746787996s) [noise=true , address=145.79.227.24:4583 , piece_id=3DZK4IUEQOT2ISCZLBSFUTHFW4JBYMGOODDLKQLKRPFFDJX7EUNQ, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:26.877124775 (601h51m26.74820782s) [noise=true , address=145.79.227.42:4122 , piece_id=LKOQTNEZDMR6IP5GXJFN74HMIWDJSGJLQ5IBLGFUH5VNQ5J6OXQA, seg_pos={0 0}, upload=527506]: Uploading piece...
Dec 30 17:20:28.568769113 (601h51m28.439852148s) [upload=527506]: Next returned nil segment
Dec 30 17:20:28.568787668 (601h51m28.439870703s) [upload=527506]: Waiting for error group managing segments...
Dec 30 17:20:28.855920496 (601h51m28.727003531s) [noise=true , address=145.79.227.8:4331 , piece_id=3D56OR643CBBW6JNXSOA7NRXQBY32OVMLRDZAMHKP5C7OM67FZFQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:28.855948720 (601h51m28.727031755s) [noise=true , address=145.79.227.24:4583 , piece_id=3DZK4IUEQOT2ISCZLBSFUTHFW4JBYMGOODDLKQLKRPFFDJX7EUNQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:28.878332862 (601h51m28.749415907s) [noise=true , address=145.79.227.51:4411 , piece_id=WEAPWVJYM3WYMGCMZCYRJWC4U3YPIPWR3D3AN3WBROHRCY35RJ6A, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:28.878374391 (601h51m28.749457426s) [noise=true , address=145.79.227.46:4601 , piece_id=P2EAZYLYUBMYAW3WLZTDZTXKTW2PNBP7CSF54LAT22MP23XPK6YQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:28.885513495 (601h51m28.75659653s) [noise=true , address=145.79.227.42:4122 , piece_id=LKOQTNEZDMR6IP5GXJFN74HMIWDJSGJLQ5IBLGFUH5VNQ5J6OXQA, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:28.937374204 (601h51m28.808457239s) [noise=true , address=145.79.227.39:4013 , piece_id=NHVSOIDKDXWCGSXZZA3J2J4LSMJCQ4OQOLUZ6NT45BDTXHC56DDQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:28.945184145 (601h51m28.81626719s) [noise=true , address=145.79.227.45:4229 , piece_id=HPRBYTHNO4OUS6N4CHJSKIGX3KIA5L2E65G4FFVEYJHNLEOVI3GA, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.033725009 (601h51m28.904808054s) [noise=true , address=77.90.31.11:4554 , piece_id=W3PMPHBXIF534CW6PYPKW4YUIMCRMX3GGNHBU57444KXGOXBQUXQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.114078672 (601h51m28.985161707s) [noise=true , address=77.90.31.14:4664 , piece_id=WHGXL4LIUJRKVET4WEM2G4467UM7X64FV5W5KBXU7DZCDARBSEUA, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.164002925 (601h51m29.03508597s) [noise=true , address=77.90.31.17:4770 , piece_id=BX5UJPEJLJJ5AA57RQRYJHG6MTGL7W6GQ3XRSATWRTUIGE2M5FPQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.216348928 (601h51m29.087431963s) [noise=true , address=45.91.116.14:4192 , piece_id=5TWFFB5AK46UWU7GYO46FCMZ7ZNORAQB7Y2AO4MXGFUKMOMC77ZA, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.219028861 (601h51m29.090111906s) [noise=true , address=77.90.31.16:4735 , piece_id=RGECA3YAS6TBWBLWIMP2G4TCG7V36RVRQDFSYTBNHU3LUG5NF3FQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.261914290 (601h51m29.132997325s) [noise=true , address=45.91.116.5:4297 , piece_id=6WC3XS7ZDCWLIFBTT3IGBV5XKQNCPSLM2NY5FVE356UBAE6PRZRQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: <nil>
Dec 30 17:20:29.261934067 (601h51m29.133017103s) [seg_pos={0 0}, upload=527506]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:20:29.262109572 (601h51m29.133192607s) [noise=true , address=77.90.31.19:4844 , piece_id=AKUW3G7O3C5UEEGHJMWRMWP72D5V3LJJPOVL5Y5QLL6KGDEALFZQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:36938->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:36938->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36938->77.90.31.19:4844: use of closed network connection
Dec 30 17:20:29.262115653 (601h51m29.133198688s) [noise=true , address=77.90.31.19:4844 , piece_id=AKUW3G7O3C5UEEGHJMWRMWP72D5V3LJJPOVL5Y5QLL6KGDEALFZQ, seg_pos={0 0}, upload=527506]: Long tail cancellation due to optimal threshold
Dec 30 17:20:29.262268104 (601h51m29.133351139s) [noise=true , address=45.91.116.4:4939 , piece_id=3WU5VRJO2LG747WQQ7DRZE7NE2BVDJ7ARU2ESSUKWZAR5SQ5H46A, seg_pos={0 0}, upload=527506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 30 17:20:29.262276971 (601h51m29.133360026s) [noise=true , address=45.91.116.4:4939 , piece_id=3WU5VRJO2LG747WQQ7DRZE7NE2BVDJ7ARU2ESSUKWZAR5SQ5H46A, seg_pos={0 0}, upload=527506]: Long tail cancellation due to optimal threshold
Dec 30 17:20:29.262921278 (601h51m29.134004323s) [noise=true , address=145.79.227.16:4909 , piece_id=WXTUOKURDVSP7K7OGB6JJVV722W6LRF7QF6MKQMEVK3WCV5HQ7GQ, seg_pos={0 0}, upload=527506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:55136->145.79.227.16:4909: use of closed network connection; context canceled
Dec 30 17:20:29.262942438 (601h51m29.134025483s) [noise=true , address=145.79.227.16:4909 , piece_id=WXTUOKURDVSP7K7OGB6JJVV722W6LRF7QF6MKQMEVK3WCV5HQ7GQ, seg_pos={0 0}, upload=527506]: Long tail cancellation due to optimal threshold
Dec 30 17:20:29.263003043 (601h51m29.134086078s) [seg_pos={0 0}, upload=527506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:20:29.263130637 (601h51m29.134213682s) [upload=527506]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:20:29.263138642 (601h51m29.134221677s) [upload=527506]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:20:29.555510955 (601h51m29.426594s) [upload=527506]: Done uploading segments...
Dec 30 17:20:29.555538788 (601h51m29.426621833s) [upload=527506]: Upload finished. err: <nil>
Duration: 30.003028531s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration:
6.050335056s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:20:50 UTC)
orbiter warning: context deadline exceeded
Duration: 6.050392755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X...X..X...........] duration:
5.852005211s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:20:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.852035869s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................X...X.................] duration:
5.905875195s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:20:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.905902006s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [X..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.X] duration:
30.003155917s trace:
https://jaeger.dev.storj.io/trace/716b24ffc5642649
FAIL (2025-12-30 17:20:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: S6R5LQNBK6MJ6QYY5WN7FXDATQUHQQ52FQ6QZDTMECWK77SCJKJQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 7GSZPBCGEFKOIQ46VVQVVJDN45N4HMMZNBVWUMPQNSNKDVIEQRMQ) context deadline exceeded
======LOGS======
Dec 30 17:19:56.362479745 (601h50m56.23356278s) [upload=527498]: Starting upload
Dec 30 17:19:56.362534359 (601h50m56.233617394s) [upload=527498]: Done starting upload
Dec 30 17:19:56.362786660 (601h50m56.233869695s) [upload=527498]: Uploading segments...
Dec 30 17:19:56.362845822 (601h50m56.233928857s) [upload=527498]: Got next segment. Inline: false
Dec 30 17:19:56.362888564 (601h50m56.233971609s) [upload=527498]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:19:56.362894836 (601h50m56.233977891s) [upload=527498]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:19:56.616396224 (601h50m56.487479259s) [seg_pos={0 0}, upload=527498]: Begin upload segment...
Dec 30 17:19:56.616479944 (601h50m56.487562979s) [seg_pos={0 0}, upload=527498]: Done begin upload segment.
Dec 30 17:19:56.616555618 (601h50m56.487638653s) [noise=true , address=45.91.116.13:4155 , piece_id=YOYDVR5LEXBBBJXRNTQRKEZOURKFLMNTVP43PETFMJLTJCBGXJSQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.616618367 (601h50m56.487701412s) [noise=true , address=145.79.227.10:4401 , piece_id=DFTGSNR6FNIQP2EZWF7ZOVMHV2DKUXYIRWWFVLCZCRX3UV5QYY6Q, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.617210425 (601h50m56.48829346s) [noise=true , address=45.91.116.4:4939 , piece_id=XGZDYPNK2L2JTYMEAAS5FBR2ASAYSRJ7CO33G5SCBGAIXPEMNS3Q, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.617637588 (601h50m56.488720633s) [noise=true , address=145.79.227.33:4869 , piece_id=3U4Y54ULFTTFGDDSA5JWEI2YDFYTXCJB4QOYGLDOMLZ3D7BPZNOQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.617675440 (601h50m56.488758485s) [noise=true , address=145.79.227.18:4006 , piece_id=E6EYH5UHHPNDK6PKP3NBBSUPQVY4F2LT6HOWZR32QJ36OXPMS6ZQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.618146297 (601h50m56.489229342s) [noise=true , address=77.90.31.14:4663 , piece_id=KAA7NQ7CQFQTRFJIRSH743RKOZKZBIREASAGCNQNN4MOSF3EF55Q, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.618567138 (601h50m56.489650183s) [noise=true , address=45.91.116.12:4225 , piece_id=S6R5LQNBK6MJ6QYY5WN7FXDATQUHQQ52FQ6QZDTMECWK77SCJKJQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.618886085 (601h50m56.48996913s) [noise=true , address=77.90.31.10:4519 , piece_id=7GSZPBCGEFKOIQ46VVQVVJDN45N4HMMZNBVWUMPQNSNKDVIEQRMQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.619331353 (601h50m56.490414398s) [noise=true , address=145.79.227.4:4186 , piece_id=YL6ZOLWMMG4FJMTK52CQFODUQFTPGTNP7243S5DJFQDVJNT3QQJA, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.619762354 (601h50m56.490845389s) [noise=true , address=45.91.116.14:4192 , piece_id=5OMCFLNRGWUV3FJT2F4AQZS4EU4PI3M6KD2OL5EWXZH3GJKDJBPA, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.619852145 (601h50m56.49093519s) [noise=true , address=145.79.227.23:4546 , piece_id=ICPELVRNEU2MOZVKKADGIF2X75ID5BIZ5UWTMRRHSI5LJMSSB26A, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.620092191 (601h50m56.491175226s) [noise=true , address=145.79.227.11:4731 , piece_id=W2FHZV7TUWW7LJKHY2RIJDN4KBLQ4DXRLE3VALXLJGKLUU7Y7OGQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.620447238 (601h50m56.491530283s) [noise=true , address=145.79.227.51:4411 , piece_id=SFYKF7HIRLCUTFZICUXP7GH6VBBZE4OBBILDLRPJJVRLUUFP2FZQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.620843132 (601h50m56.491926167s) [noise=true , address=145.79.227.15:4876 , piece_id=WH3P2RXC6PB7DWVMHHJU42MYRITXPHAZJHMZTZEHPVWJBMIZ62VQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.621170925 (601h50m56.49225396s) [noise=true , address=77.90.31.11:4554 , piece_id=NHRUREQWBRVHHOK3K6R54JOGOORG6DYTT4RJAMSPHZNYHS3ZS7DA, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:56.621522254 (601h50m56.492605299s) [noise=true , address=145.79.227.32:4835 , piece_id=YPSTTTGZZH2FGAGFR7C2LVQUCTUKABZFYGA55DI7UM4T4U22YUDQ, seg_pos={0 0}, upload=527498]: Uploading piece...
Dec 30 17:19:58.318911518 (601h50m58.189994553s) [upload=527498]: Next returned nil segment
Dec 30 17:19:58.318924834 (601h50m58.190007879s) [upload=527498]: Waiting for error group managing segments...
Dec 30 17:19:58.598523947 (601h50m58.469606982s) [noise=true , address=145.79.227.18:4006 , piece_id=E6EYH5UHHPNDK6PKP3NBBSUPQVY4F2LT6HOWZR32QJ36OXPMS6ZQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.601768896 (601h50m58.472851931s) [noise=true , address=145.79.227.4:4186 , piece_id=YL6ZOLWMMG4FJMTK52CQFODUQFTPGTNP7243S5DJFQDVJNT3QQJA, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.604713061 (601h50m58.475796107s) [noise=true , address=145.79.227.15:4876 , piece_id=WH3P2RXC6PB7DWVMHHJU42MYRITXPHAZJHMZTZEHPVWJBMIZ62VQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.616707961 (601h50m58.487791006s) [noise=true , address=145.79.227.23:4546 , piece_id=ICPELVRNEU2MOZVKKADGIF2X75ID5BIZ5UWTMRRHSI5LJMSSB26A, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.628011735 (601h50m58.49909477s) [noise=true , address=145.79.227.32:4835 , piece_id=YPSTTTGZZH2FGAGFR7C2LVQUCTUKABZFYGA55DI7UM4T4U22YUDQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.629424255 (601h50m58.5005073s) [noise=true , address=145.79.227.51:4411 , piece_id=SFYKF7HIRLCUTFZICUXP7GH6VBBZE4OBBILDLRPJJVRLUUFP2FZQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.639585693 (601h50m58.510668728s) [noise=true , address=145.79.227.10:4401 , piece_id=DFTGSNR6FNIQP2EZWF7ZOVMHV2DKUXYIRWWFVLCZCRX3UV5QYY6Q, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.672861796 (601h50m58.543944831s) [noise=true , address=145.79.227.11:4731 , piece_id=W2FHZV7TUWW7LJKHY2RIJDN4KBLQ4DXRLE3VALXLJGKLUU7Y7OGQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.800983736 (601h50m58.672066781s) [noise=true , address=77.90.31.14:4663 , piece_id=KAA7NQ7CQFQTRFJIRSH743RKOZKZBIREASAGCNQNN4MOSF3EF55Q, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.908038712 (601h50m58.779121757s) [noise=true , address=77.90.31.11:4554 , piece_id=NHRUREQWBRVHHOK3K6R54JOGOORG6DYTT4RJAMSPHZNYHS3ZS7DA, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.920310029 (601h50m58.791393074s) [noise=true , address=45.91.116.12:4225 , piece_id=S6R5LQNBK6MJ6QYY5WN7FXDATQUHQQ52FQ6QZDTMECWK77SCJKJQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:58.968875984 (601h50m58.839959019s) [noise=true , address=77.90.31.10:4519 , piece_id=7GSZPBCGEFKOIQ46VVQVVJDN45N4HMMZNBVWUMPQNSNKDVIEQRMQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:59.009075969 (601h50m58.880159004s) [noise=true , address=45.91.116.13:4155 , piece_id=YOYDVR5LEXBBBJXRNTQRKEZOURKFLMNTVP43PETFMJLTJCBGXJSQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: <nil>
Dec 30 17:19:59.009093162 (601h50m58.880176197s) [seg_pos={0 0}, upload=527498]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:19:59.009248418 (601h50m58.880331453s) [noise=true , address=145.79.227.33:4869 , piece_id=3U4Y54ULFTTFGDDSA5JWEI2YDFYTXCJB4QOYGLDOMLZ3D7BPZNOQ, seg_pos={0 0}, upload=527498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 30 17:19:59.009317399 (601h50m58.880400454s) [noise=true , address=145.79.227.33:4869 , piece_id=3U4Y54ULFTTFGDDSA5JWEI2YDFYTXCJB4QOYGLDOMLZ3D7BPZNOQ, seg_pos={0 0}, upload=527498]: Long tail cancellation due to optimal threshold
Dec 30 17:19:59.009576071 (601h50m58.880659106s) [noise=true , address=45.91.116.4:4939 , piece_id=XGZDYPNK2L2JTYMEAAS5FBR2ASAYSRJ7CO33G5SCBGAIXPEMNS3Q, seg_pos={0 0}, upload=527498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 30 17:19:59.009583306 (601h50m58.880666341s) [noise=true , address=45.91.116.4:4939 , piece_id=XGZDYPNK2L2JTYMEAAS5FBR2ASAYSRJ7CO33G5SCBGAIXPEMNS3Q, seg_pos={0 0}, upload=527498]: Long tail cancellation due to optimal threshold
Dec 30 17:19:59.009694267 (601h50m58.880777302s) [noise=true , address=45.91.116.14:4192 , piece_id=5OMCFLNRGWUV3FJT2F4AQZS4EU4PI3M6KD2OL5EWXZH3GJKDJBPA, seg_pos={0 0}, upload=527498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 30 17:19:59.009700238 (601h50m58.880783273s) [noise=true , address=45.91.116.14:4192 , piece_id=5OMCFLNRGWUV3FJT2F4AQZS4EU4PI3M6KD2OL5EWXZH3GJKDJBPA, seg_pos={0 0}, upload=527498]: Long tail cancellation due to optimal threshold
Dec 30 17:19:59.009720937 (601h50m58.880803972s) [seg_pos={0 0}, upload=527498]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:19:59.009871194 (601h50m58.880954259s) [upload=527498]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:19:59.009880241 (601h50m58.880963336s) [upload=527498]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:19:59.299875780 (601h50m59.170958825s) [upload=527498]: Done uploading segments...
Dec 30 17:19:59.299896349 (601h50m59.170979394s) [upload=527498]: Upload finished. err: <nil>
Duration: 30.003179353s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.XXX..X...XX......X.XX................X..X....X.XX..X..X....] duration:
3.619183864s trace:
https://jaeger.dev.storj.io/trace/65d425bd21adcb47
FAIL (2025-12-30 17:20:13 UTC)
orbiter warning: context deadline exceeded
Duration: 3.619210834s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X.......X................................X..........] duration:
24.595474121s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:20:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 24.595490923s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X...X..X..........] duration:
8.277012072s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:20:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.277063512s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X...X................] duration:
6.285950751s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:20:00 UTC)
orbiter warning: context deadline exceeded
Duration: 6.285988655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X.......X................................X.........] duration:
45.335522514s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:19:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 45.335550497s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X...X..X.........] duration:
29.737524747s trace:
https://jaeger.dev.storj.io/trace/18d14e6a660d59e3
FAIL (2025-12-30 17:19:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.737566878s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [..XXX..X...XX......X.XX................X..X....X.XX..X..X...] duration:
3.54999082s trace:
https://jaeger.dev.storj.io/trace/65d425bd21adcb47
FAIL (2025-12-30 17:19:24 UTC)
orbiter warning: context deadline exceeded
Duration: 3.550004726s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [XX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.] duration:
6.011911422s trace:
https://jaeger.dev.storj.io/trace/709c066c7564b22d
FAIL (2025-12-30 17:19:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: BPZAMORHKTJ5OGMOUR4FTLOQE442AW7O6P5SNJPCHK6S3UJS2BSA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Q5VRNI43ZAGV7IYM5GSXKG2STX44SBPYHG5FNCS3NZWIJ752UYVQ) context deadline exceeded
======LOGS======
Dec 30 17:18:27.541163726 (601h49m27.412246761s) [upload=527477]: Starting upload
Dec 30 17:18:27.541224571 (601h49m27.412307606s) [upload=527477]: Done starting upload
Dec 30 17:18:27.541322457 (601h49m27.412405492s) [upload=527477]: Uploading segments...
Dec 30 17:18:27.541487412 (601h49m27.412570447s) [upload=527477]: Got next segment. Inline: false
Dec 30 17:18:27.541520555 (601h49m27.41260359s) [upload=527477]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:18:27.541525755 (601h49m27.41260879s) [upload=527477]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:18:27.785642204 (601h49m27.656725249s) [seg_pos={0 0}, upload=527477]: Begin upload segment...
Dec 30 17:18:27.785697199 (601h49m27.656780244s) [seg_pos={0 0}, upload=527477]: Done begin upload segment.
Dec 30 17:18:27.785729330 (601h49m27.656812365s) [noise=true , address=45.91.116.5:4299 , piece_id=Q5VRNI43ZAGV7IYM5GSXKG2STX44SBPYHG5FNCS3NZWIJ752UYVQ, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.785824581 (601h49m27.656907616s) [noise=true , address=77.90.31.10:4520 , piece_id=X42MGNHHCYWCLVP5FA7AN5ULVF6JHGAJ5KY4BKZYDXEZ6H7PHGIA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.786354841 (601h49m27.657437876s) [noise=true , address=45.91.116.9:4264 , piece_id=GIGYBH3323FYV6KEUXGRJBXMKMXQUIN6OJWPH34SDYKHBS72HRMA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.786649001 (601h49m27.657732036s) [noise=true , address=145.79.227.48:4302 , piece_id=LFKL6S3NKE4KXTJ62CBRHMMV2FWY6NZOPIA6TUOYSQKCISZL4ALA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.786928623 (601h49m27.658011658s) [noise=true , address=145.79.227.4:4186 , piece_id=IAGSYJWGGJZ2VQEB5HGP5AJTID2UCMCYFHUI4AOORKE3UK3UIPYA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.787218465 (601h49m27.6583015s) [noise=true , address=145.79.227.5:4221 , piece_id=KKFB563X2MOYMWGBKMZY6JHZEJCRJVCMVLZWDCLOHALVVNMHZDTA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.787828778 (601h49m27.658911823s) [noise=true , address=45.91.116.10:4082 , piece_id=EFQ6WQOYPK7ZGIFVDW3UIAHWLVNWR5PHCC4VTLM7HJB3UNLB5IMA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.788211877 (601h49m27.659294912s) [noise=true , address=145.79.227.39:4014 , piece_id=5IWLEGN66INNIIDB27IPTWSKSZLOXIPMAIYHJIZ62RYGIQPBEORQ, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.788674347 (601h49m27.659757382s) [noise=true , address=145.79.227.20:4437 , piece_id=V6B6QX3IEALP5LVUGHKZJ4GW75K26FJTXNN5RXT3BDHARD5JWCOA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.789652730 (601h49m27.660735765s) [noise=true , address=145.79.227.37:4567 , piece_id=4XOYREM6YJD6ETWNTO5PFUSROK5VH52X7INZTATRZJSWGSIW27ZQ, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.790162411 (601h49m27.661245456s) [noise=true , address=145.79.227.16:4911 , piece_id=FP2KG3SNX2FUGPFGJ5ZWKKQEOAI6GUKLUVWAEULMFBFJDIVA5KHQ, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.793231374 (601h49m27.664314419s) [noise=true , address=145.79.227.19:4078 , piece_id=5J25XPZL7BYOSVB2D4RQFCSML2UDNU6WNSFVRJSH3KSTMBLRLZFA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.796497072 (601h49m27.667580108s) [noise=true , address=145.79.227.43:4157 , piece_id=AIX46KFVJZVG5PMQ5LLMX5YLFWLY4OFBB7PM53QN24PNGXDDARFQ, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.799291453 (601h49m27.670374498s) [noise=true , address=145.79.227.30:4762 , piece_id=35Q7LKBNOMS4T34FBUUOS4VXHBKJKDYJHK5X4VMPQVPHGNU5ICKQ, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.803794959 (601h49m27.674878004s) [noise=true , address=45.91.116.13:4153 , piece_id=BPZAMORHKTJ5OGMOUR4FTLOQE442AW7O6P5SNJPCHK6S3UJS2BSA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:27.805234460 (601h49m27.676317505s) [noise=true , address=145.79.227.21:4475 , piece_id=CZBJMUSOOE4Z4LTTZ2PF2N2ALHN3UOCPIN637XC2TOKPGM6PNLWA, seg_pos={0 0}, upload=527477]: Uploading piece...
Dec 30 17:18:29.497300182 (601h49m29.368383227s) [upload=527477]: Next returned nil segment
Dec 30 17:18:29.497323055 (601h49m29.36840609s) [upload=527477]: Waiting for error group managing segments...
Dec 30 17:18:29.773369783 (601h49m29.644452818s) [noise=true , address=145.79.227.48:4302 , piece_id=LFKL6S3NKE4KXTJ62CBRHMMV2FWY6NZOPIA6TUOYSQKCISZL4ALA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.792617413 (601h49m29.663700448s) [noise=true , address=145.79.227.4:4186 , piece_id=IAGSYJWGGJZ2VQEB5HGP5AJTID2UCMCYFHUI4AOORKE3UK3UIPYA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.798351803 (601h49m29.669434838s) [noise=true , address=145.79.227.16:4911 , piece_id=FP2KG3SNX2FUGPFGJ5ZWKKQEOAI6GUKLUVWAEULMFBFJDIVA5KHQ, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.798393973 (601h49m29.669477008s) [noise=true , address=145.79.227.20:4437 , piece_id=V6B6QX3IEALP5LVUGHKZJ4GW75K26FJTXNN5RXT3BDHARD5JWCOA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.807906296 (601h49m29.678989331s) [noise=true , address=145.79.227.5:4221 , piece_id=KKFB563X2MOYMWGBKMZY6JHZEJCRJVCMVLZWDCLOHALVVNMHZDTA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.842820897 (601h49m29.713903942s) [noise=true , address=145.79.227.19:4078 , piece_id=5J25XPZL7BYOSVB2D4RQFCSML2UDNU6WNSFVRJSH3KSTMBLRLZFA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.872215799 (601h49m29.743298834s) [noise=true , address=145.79.227.39:4014 , piece_id=5IWLEGN66INNIIDB27IPTWSKSZLOXIPMAIYHJIZ62RYGIQPBEORQ, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.874789419 (601h49m29.745872454s) [noise=true , address=145.79.227.37:4567 , piece_id=4XOYREM6YJD6ETWNTO5PFUSROK5VH52X7INZTATRZJSWGSIW27ZQ, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:29.881440334 (601h49m29.752523379s) [noise=true , address=145.79.227.21:4475 , piece_id=CZBJMUSOOE4Z4LTTZ2PF2N2ALHN3UOCPIN637XC2TOKPGM6PNLWA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:30.049881527 (601h49m29.920964562s) [noise=true , address=77.90.31.10:4520 , piece_id=X42MGNHHCYWCLVP5FA7AN5ULVF6JHGAJ5KY4BKZYDXEZ6H7PHGIA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:30.104030373 (601h49m29.975113409s) [noise=true , address=45.91.116.13:4153 , piece_id=BPZAMORHKTJ5OGMOUR4FTLOQE442AW7O6P5SNJPCHK6S3UJS2BSA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:30.251307399 (601h49m30.122390434s) [noise=true , address=45.91.116.5:4299 , piece_id=Q5VRNI43ZAGV7IYM5GSXKG2STX44SBPYHG5FNCS3NZWIJ752UYVQ, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:30.556511476 (601h49m30.427594511s) [noise=true , address=45.91.116.10:4082 , piece_id=EFQ6WQOYPK7ZGIFVDW3UIAHWLVNWR5PHCC4VTLM7HJB3UNLB5IMA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: <nil>
Dec 30 17:18:30.556541162 (601h49m30.427624197s) [seg_pos={0 0}, upload=527477]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:18:30.556765550 (601h49m30.427848595s) [noise=true , address=145.79.227.30:4762 , piece_id=35Q7LKBNOMS4T34FBUUOS4VXHBKJKDYJHK5X4VMPQVPHGNU5ICKQ, seg_pos={0 0}, upload=527477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 30 17:18:30.556772773 (601h49m30.427855808s) [noise=true , address=145.79.227.30:4762 , piece_id=35Q7LKBNOMS4T34FBUUOS4VXHBKJKDYJHK5X4VMPQVPHGNU5ICKQ, seg_pos={0 0}, upload=527477]: Long tail cancellation due to optimal threshold
Dec 30 17:18:30.557048428 (601h49m30.428131463s) [noise=true , address=45.91.116.9:4264 , piece_id=GIGYBH3323FYV6KEUXGRJBXMKMXQUIN6OJWPH34SDYKHBS72HRMA, seg_pos={0 0}, upload=527477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 30 17:18:30.557096810 (601h49m30.428179845s) [noise=true , address=45.91.116.9:4264 , piece_id=GIGYBH3323FYV6KEUXGRJBXMKMXQUIN6OJWPH34SDYKHBS72HRMA, seg_pos={0 0}, upload=527477]: Long tail cancellation due to optimal threshold
Dec 30 17:18:30.557119774 (601h49m30.428202819s) [noise=true , address=145.79.227.43:4157 , piece_id=AIX46KFVJZVG5PMQ5LLMX5YLFWLY4OFBB7PM53QN24PNGXDDARFQ, seg_pos={0 0}, upload=527477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 30 17:18:30.557128240 (601h49m30.428211285s) [noise=true , address=145.79.227.43:4157 , piece_id=AIX46KFVJZVG5PMQ5LLMX5YLFWLY4OFBB7PM53QN24PNGXDDARFQ, seg_pos={0 0}, upload=527477]: Long tail cancellation due to optimal threshold
Dec 30 17:18:30.557181532 (601h49m30.428264567s) [seg_pos={0 0}, upload=527477]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:18:30.557367165 (601h49m30.4284502s) [upload=527477]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:18:30.557376633 (601h49m30.428459678s) [upload=527477]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:18:30.843159913 (601h49m30.714242958s) [upload=527477]: Done uploading segments...
Dec 30 17:18:30.843185803 (601h49m30.714268848s) [upload=527477]: Upload finished. err: <nil>
Duration: 6.0119308s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................X...X...............] duration:
10.029899376s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:18:58 UTC)
orbiter warning: context deadline exceeded
Duration: 10.029925404s
core
uplink
us-select-1
us1-placement-20
|