Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
7.063520335s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:38:56 UTC)
orbiter warning: context deadline exceeded
Duration: 7.06354429s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X..X.....X.XXX..X...XX.............X..X....X.........X.X....] duration:
5.679480817s trace:
https://jaeger.dev.storj.io/trace/9dbf75abeaa89be
FAIL (2025-12-27 14:38:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:35:20.672412744 (527h6m20.543495789s) [upload=462335]: Starting upload
Dec 27 14:35:20.672491964 (527h6m20.543574999s) [upload=462335]: Done starting upload
Dec 27 14:35:20.672541790 (527h6m20.543624825s) [upload=462335]: Uploading segments...
Dec 27 14:35:20.672682818 (527h6m20.543765863s) [upload=462335]: Got next segment. Inline: false
Dec 27 14:35:20.672712986 (527h6m20.543796031s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:35:20.672717674 (527h6m20.54380072s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:35:20.914681274 (527h6m20.785764309s) [seg_pos={0 0}, upload=462335]: Begin upload segment...
Dec 27 14:35:20.914726189 (527h6m20.785809234s) [seg_pos={0 0}, upload=462335]: Done begin upload segment.
Dec 27 14:35:20.914752169 (527h6m20.785835204s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.914890172 (527h6m20.785973207s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916436556 (527h6m20.787519601s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916678387 (527h6m20.787761422s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916949904 (527h6m20.788032939s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917084160 (527h6m20.788167205s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917303557 (527h6m20.788386592s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917489933 (527h6m20.788572978s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917658985 (527h6m20.78874202s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918073984 (527h6m20.789157019s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918151182 (527h6m20.789234217s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918510335 (527h6m20.78959338s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918571973 (527h6m20.789655008s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918917842 (527h6m20.790000887s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918990981 (527h6m20.790074016s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.919355625 (527h6m20.79043866s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:22.524007669 (527h6m22.395090714s) [upload=462335]: Next returned nil segment
Dec 27 14:35:22.524025121 (527h6m22.395108156s) [upload=462335]: Waiting for error group managing segments...
Dec 27 14:35:22.909660227 (527h6m22.780743262s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.913680663 (527h6m22.784763698s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.932357266 (527h6m22.803440311s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.939360762 (527h6m22.810443807s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.943781590 (527h6m22.814864635s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.944042217 (527h6m22.815125252s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.945735902 (527h6m22.816818947s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.946767917 (527h6m22.817850962s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.948638198 (527h6m22.819721243s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.183921318 (527h6m23.055004363s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.208465163 (527h6m23.079548208s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.289781053 (527h6m23.160864088s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116613464 (527h6m23.987696509s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116636648 (527h6m23.987719693s) [seg_pos={0 0}, upload=462335]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:35:24.116917964 (527h6m23.988000999s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116936849 (527h6m23.988019894s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.116951177 (527h6m23.988034212s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116960124 (527h6m23.988043159s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117070143 (527h6m23.988153178s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.117080372 (527h6m23.988163407s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117113225 (527h6m23.98819626s) [seg_pos={0 0}, upload=462335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:35:24.117209168 (527h6m23.988292203s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:35:24.117216421 (527h6m23.988299467s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:35:24.408143527 (527h6m24.279226573s) [upload=462335]: Done uploading segments...
Dec 27 14:35:24.408169015 (527h6m24.27925206s) [upload=462335]: Upload finished. err: <nil>
Duration: 5.679506366s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..] duration:
5.001017788s trace:
https://jaeger.dev.storj.io/trace/374e9f35f69c49e4
FAIL (2025-12-27 14:38:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001044448s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X.............X..X..........] duration:
56.048939468s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:38:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 56.048955298s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
16.416012078s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:38:26 UTC)
orbiter warning: context deadline exceeded
Duration: 16.416043868s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
18.481273851s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:38:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 18.481303999s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X..X.....X.XXX..X...XX.............X..X....X.........X.X...] duration:
5.920691484s trace:
https://jaeger.dev.storj.io/trace/9dbf75abeaa89be
FAIL (2025-12-27 14:38:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:35:20.672412744 (527h6m20.543495789s) [upload=462335]: Starting upload
Dec 27 14:35:20.672491964 (527h6m20.543574999s) [upload=462335]: Done starting upload
Dec 27 14:35:20.672541790 (527h6m20.543624825s) [upload=462335]: Uploading segments...
Dec 27 14:35:20.672682818 (527h6m20.543765863s) [upload=462335]: Got next segment. Inline: false
Dec 27 14:35:20.672712986 (527h6m20.543796031s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:35:20.672717674 (527h6m20.54380072s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:35:20.914681274 (527h6m20.785764309s) [seg_pos={0 0}, upload=462335]: Begin upload segment...
Dec 27 14:35:20.914726189 (527h6m20.785809234s) [seg_pos={0 0}, upload=462335]: Done begin upload segment.
Dec 27 14:35:20.914752169 (527h6m20.785835204s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.914890172 (527h6m20.785973207s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916436556 (527h6m20.787519601s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916678387 (527h6m20.787761422s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916949904 (527h6m20.788032939s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917084160 (527h6m20.788167205s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917303557 (527h6m20.788386592s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917489933 (527h6m20.788572978s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917658985 (527h6m20.78874202s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918073984 (527h6m20.789157019s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918151182 (527h6m20.789234217s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918510335 (527h6m20.78959338s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918571973 (527h6m20.789655008s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918917842 (527h6m20.790000887s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918990981 (527h6m20.790074016s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.919355625 (527h6m20.79043866s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:22.524007669 (527h6m22.395090714s) [upload=462335]: Next returned nil segment
Dec 27 14:35:22.524025121 (527h6m22.395108156s) [upload=462335]: Waiting for error group managing segments...
Dec 27 14:35:22.909660227 (527h6m22.780743262s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.913680663 (527h6m22.784763698s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.932357266 (527h6m22.803440311s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.939360762 (527h6m22.810443807s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.943781590 (527h6m22.814864635s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.944042217 (527h6m22.815125252s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.945735902 (527h6m22.816818947s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.946767917 (527h6m22.817850962s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.948638198 (527h6m22.819721243s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.183921318 (527h6m23.055004363s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.208465163 (527h6m23.079548208s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.289781053 (527h6m23.160864088s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116613464 (527h6m23.987696509s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116636648 (527h6m23.987719693s) [seg_pos={0 0}, upload=462335]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:35:24.116917964 (527h6m23.988000999s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116936849 (527h6m23.988019894s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.116951177 (527h6m23.988034212s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116960124 (527h6m23.988043159s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117070143 (527h6m23.988153178s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.117080372 (527h6m23.988163407s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117113225 (527h6m23.98819626s) [seg_pos={0 0}, upload=462335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:35:24.117209168 (527h6m23.988292203s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:35:24.117216421 (527h6m23.988299467s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:35:24.408143527 (527h6m24.279226573s) [upload=462335]: Done uploading segments...
Dec 27 14:35:24.408169015 (527h6m24.27925206s) [upload=462335]: Upload finished. err: <nil>
Duration: 5.920712243s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X.............X..X.........] duration:
57.850602827s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:37:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 57.85063065s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX.] duration:
4.386928636s trace:
https://jaeger.dev.storj.io/trace/374e9f35f69c49e4
FAIL (2025-12-27 14:37:45 UTC)
orbiter warning: context deadline exceeded
Duration: 4.38695192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
16.711571584s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:37:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 16.711647038s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX] duration:
5.000691736s trace:
https://jaeger.dev.storj.io/trace/374e9f35f69c49e4
FAIL (2025-12-27 14:37:21 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000716324s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
15.666691072s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:37:20 UTC)
orbiter warning: context deadline exceeded
Duration: 15.666711612s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXX] duration:
5.00084093s trace:
https://jaeger.dev.storj.io/trace/2959d348ff1d526e
FAIL (2025-12-27 14:37:03 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000861608s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X..X.....X.XXX..X...XX.............X..X....X.........X.X..] duration:
5.653849701s trace:
https://jaeger.dev.storj.io/trace/9dbf75abeaa89be
FAIL (2025-12-27 14:36:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:35:20.672412744 (527h6m20.543495789s) [upload=462335]: Starting upload
Dec 27 14:35:20.672491964 (527h6m20.543574999s) [upload=462335]: Done starting upload
Dec 27 14:35:20.672541790 (527h6m20.543624825s) [upload=462335]: Uploading segments...
Dec 27 14:35:20.672682818 (527h6m20.543765863s) [upload=462335]: Got next segment. Inline: false
Dec 27 14:35:20.672712986 (527h6m20.543796031s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:35:20.672717674 (527h6m20.54380072s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:35:20.914681274 (527h6m20.785764309s) [seg_pos={0 0}, upload=462335]: Begin upload segment...
Dec 27 14:35:20.914726189 (527h6m20.785809234s) [seg_pos={0 0}, upload=462335]: Done begin upload segment.
Dec 27 14:35:20.914752169 (527h6m20.785835204s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.914890172 (527h6m20.785973207s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916436556 (527h6m20.787519601s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916678387 (527h6m20.787761422s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916949904 (527h6m20.788032939s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917084160 (527h6m20.788167205s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917303557 (527h6m20.788386592s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917489933 (527h6m20.788572978s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917658985 (527h6m20.78874202s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918073984 (527h6m20.789157019s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918151182 (527h6m20.789234217s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918510335 (527h6m20.78959338s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918571973 (527h6m20.789655008s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918917842 (527h6m20.790000887s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918990981 (527h6m20.790074016s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.919355625 (527h6m20.79043866s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:22.524007669 (527h6m22.395090714s) [upload=462335]: Next returned nil segment
Dec 27 14:35:22.524025121 (527h6m22.395108156s) [upload=462335]: Waiting for error group managing segments...
Dec 27 14:35:22.909660227 (527h6m22.780743262s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.913680663 (527h6m22.784763698s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.932357266 (527h6m22.803440311s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.939360762 (527h6m22.810443807s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.943781590 (527h6m22.814864635s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.944042217 (527h6m22.815125252s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.945735902 (527h6m22.816818947s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.946767917 (527h6m22.817850962s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.948638198 (527h6m22.819721243s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.183921318 (527h6m23.055004363s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.208465163 (527h6m23.079548208s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.289781053 (527h6m23.160864088s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116613464 (527h6m23.987696509s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116636648 (527h6m23.987719693s) [seg_pos={0 0}, upload=462335]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:35:24.116917964 (527h6m23.988000999s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116936849 (527h6m23.988019894s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.116951177 (527h6m23.988034212s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116960124 (527h6m23.988043159s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117070143 (527h6m23.988153178s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.117080372 (527h6m23.988163407s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117113225 (527h6m23.98819626s) [seg_pos={0 0}, upload=462335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:35:24.117209168 (527h6m23.988292203s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:35:24.117216421 (527h6m23.988299467s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:35:24.408143527 (527h6m24.279226573s) [upload=462335]: Done uploading segments...
Dec 27 14:35:24.408169015 (527h6m24.27925206s) [upload=462335]: Upload finished. err: <nil>
Duration: 5.653879258s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXX] duration:
5.001850543s trace:
https://jaeger.dev.storj.io/trace/53d38dace931d909
FAIL (2025-12-27 14:36:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001876312s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.................................X.............X..X........] duration:
49.421605527s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:36:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 49.421645633s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
7.349238725s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:36:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 7.34926844s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X..X..X.....X.XXX..X...XX.............X..X....X.........X.X.] duration:
5.938390286s trace:
https://jaeger.dev.storj.io/trace/9dbf75abeaa89be
FAIL (2025-12-27 14:36:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:35:20.672412744 (527h6m20.543495789s) [upload=462335]: Starting upload
Dec 27 14:35:20.672491964 (527h6m20.543574999s) [upload=462335]: Done starting upload
Dec 27 14:35:20.672541790 (527h6m20.543624825s) [upload=462335]: Uploading segments...
Dec 27 14:35:20.672682818 (527h6m20.543765863s) [upload=462335]: Got next segment. Inline: false
Dec 27 14:35:20.672712986 (527h6m20.543796031s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:35:20.672717674 (527h6m20.54380072s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:35:20.914681274 (527h6m20.785764309s) [seg_pos={0 0}, upload=462335]: Begin upload segment...
Dec 27 14:35:20.914726189 (527h6m20.785809234s) [seg_pos={0 0}, upload=462335]: Done begin upload segment.
Dec 27 14:35:20.914752169 (527h6m20.785835204s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.914890172 (527h6m20.785973207s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916436556 (527h6m20.787519601s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916678387 (527h6m20.787761422s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916949904 (527h6m20.788032939s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917084160 (527h6m20.788167205s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917303557 (527h6m20.788386592s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917489933 (527h6m20.788572978s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917658985 (527h6m20.78874202s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918073984 (527h6m20.789157019s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918151182 (527h6m20.789234217s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918510335 (527h6m20.78959338s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918571973 (527h6m20.789655008s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918917842 (527h6m20.790000887s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918990981 (527h6m20.790074016s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.919355625 (527h6m20.79043866s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:22.524007669 (527h6m22.395090714s) [upload=462335]: Next returned nil segment
Dec 27 14:35:22.524025121 (527h6m22.395108156s) [upload=462335]: Waiting for error group managing segments...
Dec 27 14:35:22.909660227 (527h6m22.780743262s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.913680663 (527h6m22.784763698s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.932357266 (527h6m22.803440311s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.939360762 (527h6m22.810443807s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.943781590 (527h6m22.814864635s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.944042217 (527h6m22.815125252s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.945735902 (527h6m22.816818947s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.946767917 (527h6m22.817850962s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.948638198 (527h6m22.819721243s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.183921318 (527h6m23.055004363s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.208465163 (527h6m23.079548208s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.289781053 (527h6m23.160864088s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116613464 (527h6m23.987696509s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116636648 (527h6m23.987719693s) [seg_pos={0 0}, upload=462335]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:35:24.116917964 (527h6m23.988000999s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116936849 (527h6m23.988019894s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.116951177 (527h6m23.988034212s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116960124 (527h6m23.988043159s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117070143 (527h6m23.988153178s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.117080372 (527h6m23.988163407s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117113225 (527h6m23.98819626s) [seg_pos={0 0}, upload=462335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:35:24.117209168 (527h6m23.988292203s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:35:24.117216421 (527h6m23.988299467s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:35:24.408143527 (527h6m24.279226573s) [upload=462335]: Done uploading segments...
Dec 27 14:35:24.408169015 (527h6m24.27925206s) [upload=462335]: Upload finished. err: <nil>
Duration: 5.938416154s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXX] duration:
5.000841471s trace:
https://jaeger.dev.storj.io/trace/47429bd5412f272f
FAIL (2025-12-27 14:36:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000865366s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
14.650241235s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:36:12 UTC)
orbiter warning: context deadline exceeded
Duration: 14.650268387s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX.................................X.............X..X.......] duration:
54.662714274s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:35:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 54.662744803s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXX] duration:
5.001170537s trace:
https://jaeger.dev.storj.io/trace/4191bf12fa12256e
FAIL (2025-12-27 14:35:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001194593s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XX..X..X.....X.XXX..X...XX.............X..X....X.........X.X] duration:
30.002911864s trace:
https://jaeger.dev.storj.io/trace/9dbf75abeaa89be
FAIL (2025-12-27 14:35:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:35:20.672412744 (527h6m20.543495789s) [upload=462335]: Starting upload
Dec 27 14:35:20.672491964 (527h6m20.543574999s) [upload=462335]: Done starting upload
Dec 27 14:35:20.672541790 (527h6m20.543624825s) [upload=462335]: Uploading segments...
Dec 27 14:35:20.672682818 (527h6m20.543765863s) [upload=462335]: Got next segment. Inline: false
Dec 27 14:35:20.672712986 (527h6m20.543796031s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:35:20.672717674 (527h6m20.54380072s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:35:20.914681274 (527h6m20.785764309s) [seg_pos={0 0}, upload=462335]: Begin upload segment...
Dec 27 14:35:20.914726189 (527h6m20.785809234s) [seg_pos={0 0}, upload=462335]: Done begin upload segment.
Dec 27 14:35:20.914752169 (527h6m20.785835204s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.914890172 (527h6m20.785973207s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916436556 (527h6m20.787519601s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916678387 (527h6m20.787761422s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.916949904 (527h6m20.788032939s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917084160 (527h6m20.788167205s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917303557 (527h6m20.788386592s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917489933 (527h6m20.788572978s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.917658985 (527h6m20.78874202s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918073984 (527h6m20.789157019s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918151182 (527h6m20.789234217s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918510335 (527h6m20.78959338s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918571973 (527h6m20.789655008s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918917842 (527h6m20.790000887s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.918990981 (527h6m20.790074016s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:20.919355625 (527h6m20.79043866s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Uploading piece...
Dec 27 14:35:22.524007669 (527h6m22.395090714s) [upload=462335]: Next returned nil segment
Dec 27 14:35:22.524025121 (527h6m22.395108156s) [upload=462335]: Waiting for error group managing segments...
Dec 27 14:35:22.909660227 (527h6m22.780743262s) [noise=true , address=145.79.227.13:4802 , piece_id=FAJRUHVZHG43XA6BHXHBMATCVVA3CZPFQHU3PG2BTCWV2J7S3CFA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.913680663 (527h6m22.784763698s) [noise=true , address=145.79.227.16:4912 , piece_id=BQVFB564AXQ5R6HPOKNJNSHGJJANZ4MQ7R6SRK6Y7QYAV4L6PH6A, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.932357266 (527h6m22.803440311s) [noise=true , address=145.79.227.28:4727 , piece_id=4I4ZFCSTD6KYIYO5S5BS3Q3OSHAJ4ZFKQZJTANQRK6SVLEVGM6MA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.939360762 (527h6m22.810443807s) [noise=true , address=145.79.227.49:4337 , piece_id=LTW73NEN2J4MRA74BVUGLW2NDYDCNNRCLY74L6KHKAHQWOYU4WYA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.943781590 (527h6m22.814864635s) [noise=true , address=145.79.227.42:4123 , piece_id=HVWGGKGJRLTK7CTEZ5KMMWZBK3X4NUT6BY5Q67JOZSRW2U6X75UA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.944042217 (527h6m22.815125252s) [noise=true , address=145.79.227.10:4402 , piece_id=2LTOEJP6QNBXNUW262AKT5H35KHYW2YRBH7TKJ5H37TR2J5YCCIQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.945735902 (527h6m22.816818947s) [noise=true , address=145.79.227.2:4113 , piece_id=J7UHMSBHXTKZ4H57366NQILVJDRRPENRPKA6ZVMKY6XCY76DEQWA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.946767917 (527h6m22.817850962s) [noise=true , address=145.79.227.45:4230 , piece_id=623QFUMQHERC7Q6NEVATLRIESBLSNSSPI2ECY7RVFTMFBYADODTQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:22.948638198 (527h6m22.819721243s) [noise=true , address=145.79.227.20:4438 , piece_id=MRIO66G7VO7ENSYQWBFHNT55UOQVI2LJIXQP3ZBSMTDACOFZ3G6Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.183921318 (527h6m23.055004363s) [noise=true , address=45.91.116.12:4227 , piece_id=3V7VJJ2B74MWLSLOV7QNT6SCBVEOZI32DAMBR3SATCR2IXYAGJOA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.208465163 (527h6m23.079548208s) [noise=true , address=45.91.116.14:4189 , piece_id=3VYGBRUNB6WALTP2IESA2MG2FP5KXMFMSG3FX3IPAY574FEZDBTA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:23.289781053 (527h6m23.160864088s) [noise=true , address=45.91.116.10:4083 , piece_id=YFZIWPSV7JTBIVNSQLSV6X77IFZJG4SRGGGHITCEKB6B3N5WTTMA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116613464 (527h6m23.987696509s) [noise=true , address=77.90.31.16:4736 , piece_id=PLUEJAXFEML4FFCRKE27PYEXXIQB32GT42P4GLOBP6XMAAH4RM3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: <nil>
Dec 27 14:35:24.116636648 (527h6m23.987719693s) [seg_pos={0 0}, upload=462335]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:35:24.116917964 (527h6m23.988000999s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116936849 (527h6m23.988019894s) [noise=true , address=45.91.116.6:4976 , piece_id=ID7FL6IYXCNAYAB647WM7X5MWAA56PY2E3327PBCIL5BWA4RACXQ, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.116951177 (527h6m23.988034212s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.116960124 (527h6m23.988043159s) [noise=true , address=45.91.116.5:4299 , piece_id=33MM5SCLBTH4OXDRLTRN76YHSCBAH2ZYEUXKXZVU67QBUNTFFS3Q, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117070143 (527h6m23.988153178s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 14:35:24.117080372 (527h6m23.988163407s) [noise=true , address=145.79.227.33:4869 , piece_id=2GEP3W4OYTTUOGOSCJWKNDVHV4XFHGRQSOHH5CI5OYGNBB7RSZDA, seg_pos={0 0}, upload=462335]: Long tail cancellation due to optimal threshold
Dec 27 14:35:24.117113225 (527h6m23.98819626s) [seg_pos={0 0}, upload=462335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:35:24.117209168 (527h6m23.988292203s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:35:24.117216421 (527h6m23.988299467s) [upload=462335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:35:24.408143527 (527h6m24.279226573s) [upload=462335]: Done uploading segments...
Dec 27 14:35:24.408169015 (527h6m24.27925206s) [upload=462335]: Upload finished. err: <nil>
Duration: 30.002935569s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXX] duration:
5.000823404s trace:
https://jaeger.dev.storj.io/trace/3fe80094061501f0
FAIL (2025-12-27 14:35:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000845366s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
9.602588179s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:35:29 UTC)
orbiter warning: context deadline exceeded
Duration: 9.602616474s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
10.980922864s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:35:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 10.980970044s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
7.863264857s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:34:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 7.863289946s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XX.................................X.............X..X......] duration:
47.003394149s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:34:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 47.003408795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XXX..X..X.....X.XXX..X...XX.............X..X....X.........X.] duration:
5.696618941s trace:
https://jaeger.dev.storj.io/trace/70cd41009b2cd1fa
FAIL (2025-12-27 14:34:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BOY7SKMOHQCMFFBP37PGUBUHLRA5ZDJDJQ5KRI2HREOGLLZSCZRA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UWHRYKX7BQ733EJ4EN5YIGL3CD7MWJTQ5FZ6DG55S3OXKZZ4O7ZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: N6WGVQ6GSRFYP2PQNR647WI5LWTHO3TSUKLPMCHVBVPOTGJN5GBA) context deadline exceeded
======LOGS======
Dec 27 14:33:16.739672506 (527h4m16.610755541s) [upload=462303]: Starting upload
Dec 27 14:33:16.739730005 (527h4m16.61081304s) [upload=462303]: Done starting upload
Dec 27 14:33:16.739765051 (527h4m16.610848086s) [upload=462303]: Uploading segments...
Dec 27 14:33:16.739934825 (527h4m16.61101787s) [upload=462303]: Got next segment. Inline: false
Dec 27 14:33:16.740011762 (527h4m16.611094807s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:33:16.740018134 (527h4m16.611101179s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:33:16.986973879 (527h4m16.858056924s) [seg_pos={0 0}, upload=462303]: Begin upload segment...
Dec 27 14:33:16.987028043 (527h4m16.858111088s) [seg_pos={0 0}, upload=462303]: Done begin upload segment.
Dec 27 14:33:16.987055485 (527h4m16.85813852s) [noise=true , address=145.79.227.32:4835 , piece_id=76UDULQPA6CRW356XXSW5DJMWG4WVJ6KQGZ64A6Z2NUGXI72RT4Q, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.987140537 (527h4m16.858223582s) [noise=true , address=145.79.227.38:4978 , piece_id=LQ5VIQNPXLW3VRMYOBXZLBWLURMW3YPDJRNUDAC34OEBKQS52ROQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.987687058 (527h4m16.858770093s) [noise=true , address=45.91.116.12:4225 , piece_id=3KFE6LFSAL5GH523SGIFQ5FJ3JRHKNNB6VZKHAZYDYPKP6OPMPTA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.988163175 (527h4m16.85924622s) [noise=true , address=77.90.31.12:4590 , piece_id=VYSHJWM2G5WGWTNVUY4MPB43VDMMLPIPIP2NENQ734TNP2AH4EWQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.988475018 (527h4m16.859558064s) [noise=true , address=145.79.227.4:4186 , piece_id=R225K75CESNGYVMIXD7TLCCYDVY4BGFQLNBWVNIMVXUNFJ554UYQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.988749120 (527h4m16.859832155s) [noise=true , address=145.79.227.35:4481 , piece_id=INZCPKTT3BPV5LSAJMSDAVXBIR77AGHWMCGN4VM23EPM33XJX44A, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.989102022 (527h4m16.860185057s) [noise=true , address=77.90.31.17:4773 , piece_id=IZQSTD5E2NCV2PNXSIKUIVLINJS7COQNYMUEUAB7ZUMYGT2O55RA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.989399869 (527h4m16.860482904s) [noise=true , address=145.79.227.7:4294 , piece_id=CPXPEC555UN4NOZFN2Q2ZS6HK6UNN3YQ3QXQPRGG2ZXKEO7I6YWQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.989685603 (527h4m16.860768638s) [noise=true , address=145.79.227.51:4409 , piece_id=OFR7ZSXRDUU2BNHMDMO3CVFTGOLOVAGPQRWGGPZ42OUPAESIU2EQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.990096656 (527h4m16.861179691s) [noise=true , address=45.91.116.7:4012 , piece_id=N6WGVQ6GSRFYP2PQNR647WI5LWTHO3TSUKLPMCHVBVPOTGJN5GBA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.990464807 (527h4m16.861547842s) [noise=true , address=77.90.31.15:4700 , piece_id=GQYNE5OS7FIRV3XV5MW42ODOA3QCBTNXGFM2PGGGK2RSJLQUJ3BQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.990746594 (527h4m16.861829629s) [noise=true , address=145.79.227.13:4802 , piece_id=BAB3N7BOGDQWWY5S2F7CQE5SW6WBRJFCYIMIRE3LB37NLIAGKQUQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.991128992 (527h4m16.862212027s) [noise=true , address=145.79.227.52:4446 , piece_id=JOGN2LW3KBYA6QGODG2YOY34F6F7WWWMVYE452E77ZVSQLLRTQAA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.991548310 (527h4m16.862631345s) [noise=true , address=45.91.116.6:4975 , piece_id=BOY7SKMOHQCMFFBP37PGUBUHLRA5ZDJDJQ5KRI2HREOGLLZSCZRA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.991867187 (527h4m16.862950232s) [noise=true , address=145.79.227.2:4113 , piece_id=TBMYNQ454PR6NMZOIOC3OKQBCTK56RATPP6KTDPWSM6ZEDH4CHFA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.992210762 (527h4m16.863293807s) [noise=true , address=77.90.31.18:4808 , piece_id=UWHRYKX7BQ733EJ4EN5YIGL3CD7MWJTQ5FZ6DG55S3OXKZZ4O7ZQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:18.706144527 (527h4m18.577227572s) [upload=462303]: Next returned nil segment
Dec 27 14:33:18.706159905 (527h4m18.577242941s) [upload=462303]: Waiting for error group managing segments...
Dec 27 14:33:18.991475626 (527h4m18.862558671s) [noise=true , address=145.79.227.13:4802 , piece_id=BAB3N7BOGDQWWY5S2F7CQE5SW6WBRJFCYIMIRE3LB37NLIAGKQUQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.991840320 (527h4m18.862923365s) [noise=true , address=145.79.227.2:4113 , piece_id=TBMYNQ454PR6NMZOIOC3OKQBCTK56RATPP6KTDPWSM6ZEDH4CHFA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.993570464 (527h4m18.864653499s) [noise=true , address=145.79.227.51:4409 , piece_id=OFR7ZSXRDUU2BNHMDMO3CVFTGOLOVAGPQRWGGPZ42OUPAESIU2EQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.994138846 (527h4m18.865221881s) [noise=true , address=145.79.227.38:4978 , piece_id=LQ5VIQNPXLW3VRMYOBXZLBWLURMW3YPDJRNUDAC34OEBKQS52ROQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.998380433 (527h4m18.869463478s) [noise=true , address=145.79.227.4:4186 , piece_id=R225K75CESNGYVMIXD7TLCCYDVY4BGFQLNBWVNIMVXUNFJ554UYQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.000531289 (527h4m18.871614334s) [noise=true , address=145.79.227.32:4835 , piece_id=76UDULQPA6CRW356XXSW5DJMWG4WVJ6KQGZ64A6Z2NUGXI72RT4Q, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.013980518 (527h4m18.885063553s) [noise=true , address=145.79.227.35:4481 , piece_id=INZCPKTT3BPV5LSAJMSDAVXBIR77AGHWMCGN4VM23EPM33XJX44A, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.021941237 (527h4m18.893024272s) [noise=true , address=145.79.227.7:4294 , piece_id=CPXPEC555UN4NOZFN2Q2ZS6HK6UNN3YQ3QXQPRGG2ZXKEO7I6YWQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.034891978 (527h4m18.905975013s) [noise=true , address=145.79.227.52:4446 , piece_id=JOGN2LW3KBYA6QGODG2YOY34F6F7WWWMVYE452E77ZVSQLLRTQAA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.233297937 (527h4m19.104380982s) [noise=true , address=45.91.116.7:4012 , piece_id=N6WGVQ6GSRFYP2PQNR647WI5LWTHO3TSUKLPMCHVBVPOTGJN5GBA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.312019056 (527h4m19.183102101s) [noise=true , address=77.90.31.18:4808 , piece_id=UWHRYKX7BQ733EJ4EN5YIGL3CD7MWJTQ5FZ6DG55S3OXKZZ4O7ZQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.592920189 (527h4m19.464003234s) [noise=true , address=77.90.31.12:4590 , piece_id=VYSHJWM2G5WGWTNVUY4MPB43VDMMLPIPIP2NENQ734TNP2AH4EWQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.832992598 (527h4m19.704075633s) [noise=true , address=45.91.116.6:4975 , piece_id=BOY7SKMOHQCMFFBP37PGUBUHLRA5ZDJDJQ5KRI2HREOGLLZSCZRA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.833011895 (527h4m19.70409494s) [seg_pos={0 0}, upload=462303]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:33:19.833226684 (527h4m19.704309729s) [noise=true , address=77.90.31.17:4773 , piece_id=IZQSTD5E2NCV2PNXSIKUIVLINJS7COQNYMUEUAB7ZUMYGT2O55RA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:33:19.833236052 (527h4m19.704319097s) [noise=true , address=77.90.31.17:4773 , piece_id=IZQSTD5E2NCV2PNXSIKUIVLINJS7COQNYMUEUAB7ZUMYGT2O55RA, seg_pos={0 0}, upload=462303]: Long tail cancellation due to optimal threshold
Dec 27 14:33:19.833570469 (527h4m19.704653524s) [noise=true , address=77.90.31.15:4700 , piece_id=GQYNE5OS7FIRV3XV5MW42ODOA3QCBTNXGFM2PGGGK2RSJLQUJ3BQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 27 14:33:19.833614302 (527h4m19.704697337s) [noise=true , address=77.90.31.15:4700 , piece_id=GQYNE5OS7FIRV3XV5MW42ODOA3QCBTNXGFM2PGGGK2RSJLQUJ3BQ, seg_pos={0 0}, upload=462303]: Long tail cancellation due to optimal threshold
Dec 27 14:33:19.833618210 (527h4m19.704701255s) [noise=true , address=45.91.116.12:4225 , piece_id=3KFE6LFSAL5GH523SGIFQ5FJ3JRHKNNB6VZKHAZYDYPKP6OPMPTA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:52346->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:52346->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52346->45.91.116.12:4225: use of closed network connection
Dec 27 14:33:19.833626967 (527h4m19.704710012s) [noise=true , address=45.91.116.12:4225 , piece_id=3KFE6LFSAL5GH523SGIFQ5FJ3JRHKNNB6VZKHAZYDYPKP6OPMPTA, seg_pos={0 0}, upload=462303]: Long tail cancellation due to optimal threshold
Dec 27 14:33:19.833647936 (527h4m19.704730971s) [seg_pos={0 0}, upload=462303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:33:19.834015056 (527h4m19.705098111s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:33:19.834024974 (527h4m19.705108009s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:33:20.132987861 (527h4m20.004070906s) [upload=462303]: Done uploading segments...
Dec 27 14:33:20.133061893 (527h4m20.004144938s) [upload=462303]: Upload finished. err: <nil>
Duration: 5.696640141s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXX] duration:
5.002172537s trace:
https://jaeger.dev.storj.io/trace/182cd096195b1245
FAIL (2025-12-27 14:34:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002197884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............XX..............................................] duration:
18.249726986s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:34:25 UTC)
orbiter warning: context deadline exceeded
Duration: 18.249751635s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
16.752270681s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:34:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 16.752312089s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX.................................X.............X..X.....] duration:
51.284058886s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:33:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 51.284075508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XXXX..X..X.....X.XXX..X...XX.............X..X....X.........X] duration:
30.003755012s trace:
https://jaeger.dev.storj.io/trace/70cd41009b2cd1fa
FAIL (2025-12-27 14:33:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BOY7SKMOHQCMFFBP37PGUBUHLRA5ZDJDJQ5KRI2HREOGLLZSCZRA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UWHRYKX7BQ733EJ4EN5YIGL3CD7MWJTQ5FZ6DG55S3OXKZZ4O7ZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: N6WGVQ6GSRFYP2PQNR647WI5LWTHO3TSUKLPMCHVBVPOTGJN5GBA) context deadline exceeded
======LOGS======
Dec 27 14:33:16.739672506 (527h4m16.610755541s) [upload=462303]: Starting upload
Dec 27 14:33:16.739730005 (527h4m16.61081304s) [upload=462303]: Done starting upload
Dec 27 14:33:16.739765051 (527h4m16.610848086s) [upload=462303]: Uploading segments...
Dec 27 14:33:16.739934825 (527h4m16.61101787s) [upload=462303]: Got next segment. Inline: false
Dec 27 14:33:16.740011762 (527h4m16.611094807s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:33:16.740018134 (527h4m16.611101179s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:33:16.986973879 (527h4m16.858056924s) [seg_pos={0 0}, upload=462303]: Begin upload segment...
Dec 27 14:33:16.987028043 (527h4m16.858111088s) [seg_pos={0 0}, upload=462303]: Done begin upload segment.
Dec 27 14:33:16.987055485 (527h4m16.85813852s) [noise=true , address=145.79.227.32:4835 , piece_id=76UDULQPA6CRW356XXSW5DJMWG4WVJ6KQGZ64A6Z2NUGXI72RT4Q, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.987140537 (527h4m16.858223582s) [noise=true , address=145.79.227.38:4978 , piece_id=LQ5VIQNPXLW3VRMYOBXZLBWLURMW3YPDJRNUDAC34OEBKQS52ROQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.987687058 (527h4m16.858770093s) [noise=true , address=45.91.116.12:4225 , piece_id=3KFE6LFSAL5GH523SGIFQ5FJ3JRHKNNB6VZKHAZYDYPKP6OPMPTA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.988163175 (527h4m16.85924622s) [noise=true , address=77.90.31.12:4590 , piece_id=VYSHJWM2G5WGWTNVUY4MPB43VDMMLPIPIP2NENQ734TNP2AH4EWQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.988475018 (527h4m16.859558064s) [noise=true , address=145.79.227.4:4186 , piece_id=R225K75CESNGYVMIXD7TLCCYDVY4BGFQLNBWVNIMVXUNFJ554UYQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.988749120 (527h4m16.859832155s) [noise=true , address=145.79.227.35:4481 , piece_id=INZCPKTT3BPV5LSAJMSDAVXBIR77AGHWMCGN4VM23EPM33XJX44A, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.989102022 (527h4m16.860185057s) [noise=true , address=77.90.31.17:4773 , piece_id=IZQSTD5E2NCV2PNXSIKUIVLINJS7COQNYMUEUAB7ZUMYGT2O55RA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.989399869 (527h4m16.860482904s) [noise=true , address=145.79.227.7:4294 , piece_id=CPXPEC555UN4NOZFN2Q2ZS6HK6UNN3YQ3QXQPRGG2ZXKEO7I6YWQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.989685603 (527h4m16.860768638s) [noise=true , address=145.79.227.51:4409 , piece_id=OFR7ZSXRDUU2BNHMDMO3CVFTGOLOVAGPQRWGGPZ42OUPAESIU2EQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.990096656 (527h4m16.861179691s) [noise=true , address=45.91.116.7:4012 , piece_id=N6WGVQ6GSRFYP2PQNR647WI5LWTHO3TSUKLPMCHVBVPOTGJN5GBA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.990464807 (527h4m16.861547842s) [noise=true , address=77.90.31.15:4700 , piece_id=GQYNE5OS7FIRV3XV5MW42ODOA3QCBTNXGFM2PGGGK2RSJLQUJ3BQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.990746594 (527h4m16.861829629s) [noise=true , address=145.79.227.13:4802 , piece_id=BAB3N7BOGDQWWY5S2F7CQE5SW6WBRJFCYIMIRE3LB37NLIAGKQUQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.991128992 (527h4m16.862212027s) [noise=true , address=145.79.227.52:4446 , piece_id=JOGN2LW3KBYA6QGODG2YOY34F6F7WWWMVYE452E77ZVSQLLRTQAA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.991548310 (527h4m16.862631345s) [noise=true , address=45.91.116.6:4975 , piece_id=BOY7SKMOHQCMFFBP37PGUBUHLRA5ZDJDJQ5KRI2HREOGLLZSCZRA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.991867187 (527h4m16.862950232s) [noise=true , address=145.79.227.2:4113 , piece_id=TBMYNQ454PR6NMZOIOC3OKQBCTK56RATPP6KTDPWSM6ZEDH4CHFA, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:16.992210762 (527h4m16.863293807s) [noise=true , address=77.90.31.18:4808 , piece_id=UWHRYKX7BQ733EJ4EN5YIGL3CD7MWJTQ5FZ6DG55S3OXKZZ4O7ZQ, seg_pos={0 0}, upload=462303]: Uploading piece...
Dec 27 14:33:18.706144527 (527h4m18.577227572s) [upload=462303]: Next returned nil segment
Dec 27 14:33:18.706159905 (527h4m18.577242941s) [upload=462303]: Waiting for error group managing segments...
Dec 27 14:33:18.991475626 (527h4m18.862558671s) [noise=true , address=145.79.227.13:4802 , piece_id=BAB3N7BOGDQWWY5S2F7CQE5SW6WBRJFCYIMIRE3LB37NLIAGKQUQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.991840320 (527h4m18.862923365s) [noise=true , address=145.79.227.2:4113 , piece_id=TBMYNQ454PR6NMZOIOC3OKQBCTK56RATPP6KTDPWSM6ZEDH4CHFA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.993570464 (527h4m18.864653499s) [noise=true , address=145.79.227.51:4409 , piece_id=OFR7ZSXRDUU2BNHMDMO3CVFTGOLOVAGPQRWGGPZ42OUPAESIU2EQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.994138846 (527h4m18.865221881s) [noise=true , address=145.79.227.38:4978 , piece_id=LQ5VIQNPXLW3VRMYOBXZLBWLURMW3YPDJRNUDAC34OEBKQS52ROQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:18.998380433 (527h4m18.869463478s) [noise=true , address=145.79.227.4:4186 , piece_id=R225K75CESNGYVMIXD7TLCCYDVY4BGFQLNBWVNIMVXUNFJ554UYQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.000531289 (527h4m18.871614334s) [noise=true , address=145.79.227.32:4835 , piece_id=76UDULQPA6CRW356XXSW5DJMWG4WVJ6KQGZ64A6Z2NUGXI72RT4Q, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.013980518 (527h4m18.885063553s) [noise=true , address=145.79.227.35:4481 , piece_id=INZCPKTT3BPV5LSAJMSDAVXBIR77AGHWMCGN4VM23EPM33XJX44A, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.021941237 (527h4m18.893024272s) [noise=true , address=145.79.227.7:4294 , piece_id=CPXPEC555UN4NOZFN2Q2ZS6HK6UNN3YQ3QXQPRGG2ZXKEO7I6YWQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.034891978 (527h4m18.905975013s) [noise=true , address=145.79.227.52:4446 , piece_id=JOGN2LW3KBYA6QGODG2YOY34F6F7WWWMVYE452E77ZVSQLLRTQAA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.233297937 (527h4m19.104380982s) [noise=true , address=45.91.116.7:4012 , piece_id=N6WGVQ6GSRFYP2PQNR647WI5LWTHO3TSUKLPMCHVBVPOTGJN5GBA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.312019056 (527h4m19.183102101s) [noise=true , address=77.90.31.18:4808 , piece_id=UWHRYKX7BQ733EJ4EN5YIGL3CD7MWJTQ5FZ6DG55S3OXKZZ4O7ZQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.592920189 (527h4m19.464003234s) [noise=true , address=77.90.31.12:4590 , piece_id=VYSHJWM2G5WGWTNVUY4MPB43VDMMLPIPIP2NENQ734TNP2AH4EWQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.832992598 (527h4m19.704075633s) [noise=true , address=45.91.116.6:4975 , piece_id=BOY7SKMOHQCMFFBP37PGUBUHLRA5ZDJDJQ5KRI2HREOGLLZSCZRA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: <nil>
Dec 27 14:33:19.833011895 (527h4m19.70409494s) [seg_pos={0 0}, upload=462303]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:33:19.833226684 (527h4m19.704309729s) [noise=true , address=77.90.31.17:4773 , piece_id=IZQSTD5E2NCV2PNXSIKUIVLINJS7COQNYMUEUAB7ZUMYGT2O55RA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:33:19.833236052 (527h4m19.704319097s) [noise=true , address=77.90.31.17:4773 , piece_id=IZQSTD5E2NCV2PNXSIKUIVLINJS7COQNYMUEUAB7ZUMYGT2O55RA, seg_pos={0 0}, upload=462303]: Long tail cancellation due to optimal threshold
Dec 27 14:33:19.833570469 (527h4m19.704653524s) [noise=true , address=77.90.31.15:4700 , piece_id=GQYNE5OS7FIRV3XV5MW42ODOA3QCBTNXGFM2PGGGK2RSJLQUJ3BQ, seg_pos={0 0}, upload=462303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 27 14:33:19.833614302 (527h4m19.704697337s) [noise=true , address=77.90.31.15:4700 , piece_id=GQYNE5OS7FIRV3XV5MW42ODOA3QCBTNXGFM2PGGGK2RSJLQUJ3BQ, seg_pos={0 0}, upload=462303]: Long tail cancellation due to optimal threshold
Dec 27 14:33:19.833618210 (527h4m19.704701255s) [noise=true , address=45.91.116.12:4225 , piece_id=3KFE6LFSAL5GH523SGIFQ5FJ3JRHKNNB6VZKHAZYDYPKP6OPMPTA, seg_pos={0 0}, upload=462303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:52346->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:52346->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52346->45.91.116.12:4225: use of closed network connection
Dec 27 14:33:19.833626967 (527h4m19.704710012s) [noise=true , address=45.91.116.12:4225 , piece_id=3KFE6LFSAL5GH523SGIFQ5FJ3JRHKNNB6VZKHAZYDYPKP6OPMPTA, seg_pos={0 0}, upload=462303]: Long tail cancellation due to optimal threshold
Dec 27 14:33:19.833647936 (527h4m19.704730971s) [seg_pos={0 0}, upload=462303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:33:19.834015056 (527h4m19.705098111s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:33:19.834024974 (527h4m19.705108009s) [upload=462303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:33:20.132987861 (527h4m20.004070906s) [upload=462303]: Done uploading segments...
Dec 27 14:33:20.133061893 (527h4m20.004144938s) [upload=462303]: Upload finished. err: <nil>
Duration: 30.003782464s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
8.28470926s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:33:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 8.284737003s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............XX.............................................] duration:
8.514713889s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:33:34 UTC)
orbiter warning: context deadline exceeded
Duration: 8.514742142s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXX] duration:
5.000650588s trace:
https://jaeger.dev.storj.io/trace/b41bc1dcbe134a9
FAIL (2025-12-27 14:33:19 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000672609s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............XX............................................] duration:
20.138352424s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:33:03 UTC)
orbiter warning: context deadline exceeded
Duration: 20.138376138s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XX.................................X.............X..X....] duration:
59.175120236s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:33:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 59.175150253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXX] duration:
5.001261881s trace:
https://jaeger.dev.storj.io/trace/1b302e83e6c7e529
FAIL (2025-12-27 14:32:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001286548s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
8.696711913s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:32:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 8.696750688s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
25.09413739s trace:
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:32:24 UTC)
orbiter warning: context deadline exceeded
Duration: 25.094160475s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXX] duration:
5.000302214s trace:
https://jaeger.dev.storj.io/trace/12b1027bfc978977
FAIL (2025-12-27 14:32:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000895794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XXXX..X..X.....X.XXX..X...XX.............X..X....X.........] duration:
6.003750243s trace:
https://jaeger.dev.storj.io/trace/5ae325eb6324f9df
FAIL (2025-12-27 14:32:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:25:57.726806158 (526h56m57.597889203s) [upload=462204]: Starting upload
Dec 27 14:25:57.726903824 (526h56m57.597986869s) [upload=462204]: Done starting upload
Dec 27 14:25:57.726980620 (526h56m57.598063685s) [upload=462204]: Uploading segments...
Dec 27 14:25:57.727243751 (526h56m57.598326806s) [upload=462204]: Got next segment. Inline: false
Dec 27 14:25:57.727291442 (526h56m57.598374477s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:25:57.727297493 (526h56m57.598380539s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:25:57.971809967 (526h56m57.842893002s) [seg_pos={0 0}, upload=462204]: Begin upload segment...
Dec 27 14:25:57.971842348 (526h56m57.842925383s) [seg_pos={0 0}, upload=462204]: Done begin upload segment.
Dec 27 14:25:57.971953310 (526h56m57.843036345s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.973294103 (526h56m57.844377138s) [noise=true , address=77.90.31.21:4878 , piece_id=44G3B3HDGBXWV6GDDTQQ62DV6QZOPZD6DCXFL6YKITUN2URCJPHQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.975245218 (526h56m57.846328283s) [noise=true , address=145.79.227.37:4565 , piece_id=4V3FY7ZJ33DZEVH27DAEDKTX64SGXT5XOFVXIIL4MH72SPHHJGWQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.975831965 (526h56m57.84691501s) [noise=true , address=145.79.227.45:4229 , piece_id=OXZX5MDVEKEDFENXXNRHIPQPVOFNUPYD5QF4YIRU3MYTJMK2PCSQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976249150 (526h56m57.847332185s) [noise=true , address=145.79.227.19:4078 , piece_id=LQS2DVYUHHMN6XL4ERT64FZSIMF3QZGMP4EHQERY73L3IQYJ3VCA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976656726 (526h56m57.847739761s) [noise=true , address=77.90.31.14:4662 , piece_id=HEKU46BZ64NKRGS4GTHNCVYX6KWJOMSR6J7JWUH3WZCOKGFZR3ZQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976990611 (526h56m57.848073656s) [noise=true , address=45.91.116.12:4228 , piece_id=J2EDKOABXIKKVSGZ7E3KFBOMFRXMWAFDKDUZJRALYGQYSHGCH6RQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.977447332 (526h56m57.848530367s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.977774625 (526h56m57.84885766s) [noise=true , address=45.91.116.9:4261 , piece_id=2E7LQB2AZOISLMBMICILM2H6OXC62AFTASTIJIHSU6VL4GKUR7WA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.979036097 (526h56m57.850119132s) [noise=true , address=45.91.116.10:4082 , piece_id=ESRYSZW7HJXGUQO5MEHDGTGEXLUQXMF7CKJPTATSA3AVTIEJCGJA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.981393726 (526h56m57.852476761s) [noise=true , address=145.79.227.15:4874 , piece_id=GGAUHHT6MZ76YRJ4EF4HTVMYX557SMXOL3IZJGWU7MD2I3JJ3ZTA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.982686808 (526h56m57.853769853s) [noise=true , address=145.79.227.16:4909 , piece_id=JRCQWLEITVU2FI6DPIB6RTXDXPRMLJGQEMQYSCQWS5CIWGS2D6QQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.983994349 (526h56m57.855077384s) [noise=true , address=145.79.227.10:4402 , piece_id=VIHRGMNXGF6WCUXSS2SLKH43AO73RY5WO2XNECPEN42FAOBR5Z7A, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.985345180 (526h56m57.856428215s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.986387134 (526h56m57.857470179s) [noise=true , address=45.91.116.13:4155 , piece_id=5BVUD2I2A37VTIYGCN3MUCS33ARUT6EFXVI7H4BBZKKDICI3NMLQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.986623464 (526h56m57.857706499s) [noise=true , address=77.90.31.12:4590 , piece_id=GDCIO46UXVYY2QORA5SP5ILCYB6YJ444HFOKRIGEBNGQC3FJ5FBQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:59.697992155 (526h56m59.56907519s) [upload=462204]: Next returned nil segment
Dec 27 14:25:59.698004909 (526h56m59.569087944s) [upload=462204]: Waiting for error group managing segments...
Dec 27 14:25:59.979483873 (526h56m59.850566908s) [noise=true , address=145.79.227.16:4909 , piece_id=JRCQWLEITVU2FI6DPIB6RTXDXPRMLJGQEMQYSCQWS5CIWGS2D6QQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.982814666 (526h56m59.853897711s) [noise=true , address=145.79.227.45:4229 , piece_id=OXZX5MDVEKEDFENXXNRHIPQPVOFNUPYD5QF4YIRU3MYTJMK2PCSQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.984622959 (526h56m59.855706004s) [noise=true , address=145.79.227.19:4078 , piece_id=LQS2DVYUHHMN6XL4ERT64FZSIMF3QZGMP4EHQERY73L3IQYJ3VCA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.985208203 (526h56m59.856291238s) [noise=true , address=145.79.227.10:4402 , piece_id=VIHRGMNXGF6WCUXSS2SLKH43AO73RY5WO2XNECPEN42FAOBR5Z7A, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.005223195 (526h56m59.87630623s) [noise=true , address=145.79.227.37:4565 , piece_id=4V3FY7ZJ33DZEVH27DAEDKTX64SGXT5XOFVXIIL4MH72SPHHJGWQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.010871040 (526h56m59.881954075s) [noise=true , address=145.79.227.15:4874 , piece_id=GGAUHHT6MZ76YRJ4EF4HTVMYX557SMXOL3IZJGWU7MD2I3JJ3ZTA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.237504679 (526h57m0.108587724s) [noise=true , address=45.91.116.13:4155 , piece_id=5BVUD2I2A37VTIYGCN3MUCS33ARUT6EFXVI7H4BBZKKDICI3NMLQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.273392205 (526h57m0.14447524s) [noise=true , address=77.90.31.21:4878 , piece_id=44G3B3HDGBXWV6GDDTQQ62DV6QZOPZD6DCXFL6YKITUN2URCJPHQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.340268639 (526h57m0.211351674s) [noise=true , address=45.91.116.9:4261 , piece_id=2E7LQB2AZOISLMBMICILM2H6OXC62AFTASTIJIHSU6VL4GKUR7WA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.343730832 (526h57m0.214813867s) [noise=true , address=45.91.116.10:4082 , piece_id=ESRYSZW7HJXGUQO5MEHDGTGEXLUQXMF7CKJPTATSA3AVTIEJCGJA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.820785406 (526h57m0.691868451s) [noise=true , address=77.90.31.12:4590 , piece_id=GDCIO46UXVYY2QORA5SP5ILCYB6YJ444HFOKRIGEBNGQC3FJ5FBQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.860474380 (526h57m0.731557415s) [noise=true , address=77.90.31.14:4662 , piece_id=HEKU46BZ64NKRGS4GTHNCVYX6KWJOMSR6J7JWUH3WZCOKGFZR3ZQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:01.004516601 (526h57m0.875599647s) [noise=true , address=45.91.116.12:4228 , piece_id=J2EDKOABXIKKVSGZ7E3KFBOMFRXMWAFDKDUZJRALYGQYSHGCH6RQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:01.004541869 (526h57m0.875624914s) [seg_pos={0 0}, upload=462204]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:26:01.004764193 (526h57m0.875847228s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.004772679 (526h57m0.875855714s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.004918436 (526h57m0.876001481s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.004925420 (526h57m0.876008455s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.005031813 (526h57m0.876114848s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.005048103 (526h57m0.876131148s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.005079343 (526h57m0.876162388s) [seg_pos={0 0}, upload=462204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:26:01.005175646 (526h57m0.876258682s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:26:01.005188691 (526h57m0.876271726s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:26:01.291045352 (526h57m1.162128417s) [upload=462204]: Done uploading segments...
Dec 27 14:26:01.291068015 (526h57m1.16215108s) [upload=462204]: Upload finished. err: <nil>
Duration: 6.00378043s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XX.................................X.............X..X...] duration:
48.258911701s trace:
https://jaeger.dev.storj.io/trace/658fc062bd24e27c
FAIL (2025-12-27 14:32:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151010387 (526h59m14.022093432s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151292585 (526h59m14.02237563s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151428734 (526h59m14.022511779s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151815350 (526h59m14.022898395s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.151823185 (526h59m14.02290622s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152223397 (526h59m14.023306442s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152360438 (526h59m14.023443483s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152588231 (526h59m14.023671277s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.152941385 (526h59m14.02402443s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153031907 (526h59m14.024114952s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.153499307 (526h59m14.024582352s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.154210100 (526h59m14.025293145s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.155416087 (526h59m14.026499122s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.159380887 (526h59m14.030463932s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:14.160458649 (526h59m14.031541694s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Uploading piece...
Dec 27 14:28:15.585384582 (526h59m15.456467618s) [upload=462236]: Next returned nil segment
Dec 27 14:28:15.585399080 (526h59m15.456482125s) [upload=462236]: Waiting for error group managing segments...
Dec 27 14:28:16.002882640 (526h59m15.873965686s) [noise=true , address=64.49.13.140:4505 , piece_id=IRWUETYFYC4BCP52X4JZLYROZTIR2ATPM5SSISCLDU66JHDBGBQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.029187370 (526h59m15.900270405s) [noise=true , address=145.79.227.36:4543 , piece_id=JW74CGG5L2N64VPT2DMYM7P53J5XJ56S3HDBXQVKNV6BFDQWSVQQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.030356485 (526h59m15.90143953s) [noise=true , address=145.79.227.35:4502 , piece_id=ZCXSHZGDZW4OSUKIWJ7K2CKL7ROAYWH472N2G5AOYT673DPODUUA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.230158022 (526h59m16.101241057s) [noise=true , address=77.90.31.18:4818 , piece_id=FHW7PNTLPI7DBCAM62CRQEYR2O74XZQ5LU62J4E4JSDRYR725PWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.379164486 (526h59m16.250247531s) [noise=true , address=45.91.116.6:4983 , piece_id=LOOKOGXHQS3QYPSOXHZ3JWGFR2GFNOEKKUS6A5NJXMJL6ZZBV7MA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.412418918 (526h59m16.283501953s) [noise=true , address=45.91.116.14:4213 , piece_id=SK3BZT5NV4FEVT6LDWFR3E3ASBFY6YMIKMCOGQBFUVF3FR6MJVGA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.522287246 (526h59m16.393370291s) [noise=true , address=77.90.31.17:4794 , piece_id=7JLZMUSFX2NWH2BKXFGE57ST7A4FU2ABP6AWNU2VRHBN5F7W6Q2Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.876863911 (526h59m16.747946956s) [noise=true , address=145.79.227.30:4782 , piece_id=KGITKD5OCT3QHE2NCQISVZBAWAVVEDW5HNP54UNRKT2KELZBDVWA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919227459 (526h59m16.790310504s) [noise=true , address=145.79.227.33:4888 , piece_id=65332U7BLDEABQOUIBCRDGPQVGLYFFOC5JRH2TKIHR5IS4C7XX7Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:16.919448138 (526h59m16.790531204s) [noise=true , address=145.79.227.44:4227 , piece_id=SSX5OV3YNRWFHFTET5K3TEVGZAZP7F7DPMIUY6F7IJNYMGQ4TNXQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.002653255 (526h59m16.8737363s) [noise=true , address=77.90.31.14:4671 , piece_id=KDM5NMHG6IC53KGRYZMBHTPOS5CH7TXRPZBPEB2MMTP5WSL733RQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.127759850 (526h59m16.998842885s) [noise=true , address=145.79.227.34:4937 , piece_id=63PSIEBZOGUUQXDZHXQ4C3XCOG6Q22BJKOZBMLKEECTM4ATUAD6A, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.213982673 (526h59m17.085065738s) [noise=true , address=45.91.116.5:4330 , piece_id=3OTZ2LGHKVO72OKYE3UZ56TVD2AH6IQPR3OGUI2FIS3Z6UBVJ3SA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494662826 (526h59m17.365745862s) [noise=true , address=77.90.31.12:4624 , piece_id=KPM474EJ5APB5RP74MIIAWRMZST7QJ4HPNGAXZ5P66V6UDNCA4YA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: <nil>
Dec 27 14:28:17.494681802 (526h59m17.365764847s) [seg_pos={0 0}, upload=462236]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:28:17.494909446 (526h59m17.365992481s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39880->64.49.13.169:4710: use of closed network connection
Dec 27 14:28:17.494916169 (526h59m17.365999214s) [noise=true , address=64.49.13.169:4710 , piece_id=S4MNXHMRUIZ43NKIXORDHTOKVWRJVIXMXHYDR5TD5ABE6MJO3QMA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.494999146 (526h59m17.366082181s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495017602 (526h59m17.366100647s) [noise=true , address=77.90.31.16:4760 , piece_id=TAFZQY3X35RWMKQQU5FONCXXDVYOV4HJYYGQ4IMVT5PLPASHC7TQ, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495148451 (526h59m17.366231496s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495154533 (526h59m17.366237578s) [noise=true , address=77.90.31.11:4583 , piece_id=GU7QMLXSXKJCDSKYNE4372QYDUBN2YNTUWSO7F4TXXS3BEKHNPNA, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495194709 (526h59m17.366277744s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 14:28:17.495201422 (526h59m17.366284457s) [noise=true , address=77.90.31.13:4642 , piece_id=5YCULNKUIWQXEARPJ4TLEAV7B2LWBJ3KJBSA4RAVV6HUUSBZF73Q, seg_pos={0 0}, upload=462236]: Long tail cancellation due to optimal threshold
Dec 27 14:28:17.495219146 (526h59m17.366302181s) [seg_pos={0 0}, upload=462236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:28:17.495289380 (526h59m17.366372425s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:28:17.495294870 (526h59m17.366377915s) [upload=462236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:28:17.794748701 (526h59m17.665831736s) [upload=462236]: Done uploading segments...
Dec 27 14:28:17.794769692 (526h59m17.665852737s) [upload=462236]: Upload finished. err: <nil>
Duration: 48.258947661s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
17.708325034s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:31:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 17.708361984s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.XXXX..X..X.....X.XXX..X...XX.............X..X....X........] duration:
5.645874435s trace:
https://jaeger.dev.storj.io/trace/5ae325eb6324f9df
FAIL (2025-12-27 14:31:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:25:57.726806158 (526h56m57.597889203s) [upload=462204]: Starting upload
Dec 27 14:25:57.726903824 (526h56m57.597986869s) [upload=462204]: Done starting upload
Dec 27 14:25:57.726980620 (526h56m57.598063685s) [upload=462204]: Uploading segments...
Dec 27 14:25:57.727243751 (526h56m57.598326806s) [upload=462204]: Got next segment. Inline: false
Dec 27 14:25:57.727291442 (526h56m57.598374477s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:25:57.727297493 (526h56m57.598380539s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:25:57.971809967 (526h56m57.842893002s) [seg_pos={0 0}, upload=462204]: Begin upload segment...
Dec 27 14:25:57.971842348 (526h56m57.842925383s) [seg_pos={0 0}, upload=462204]: Done begin upload segment.
Dec 27 14:25:57.971953310 (526h56m57.843036345s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.973294103 (526h56m57.844377138s) [noise=true , address=77.90.31.21:4878 , piece_id=44G3B3HDGBXWV6GDDTQQ62DV6QZOPZD6DCXFL6YKITUN2URCJPHQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.975245218 (526h56m57.846328283s) [noise=true , address=145.79.227.37:4565 , piece_id=4V3FY7ZJ33DZEVH27DAEDKTX64SGXT5XOFVXIIL4MH72SPHHJGWQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.975831965 (526h56m57.84691501s) [noise=true , address=145.79.227.45:4229 , piece_id=OXZX5MDVEKEDFENXXNRHIPQPVOFNUPYD5QF4YIRU3MYTJMK2PCSQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976249150 (526h56m57.847332185s) [noise=true , address=145.79.227.19:4078 , piece_id=LQS2DVYUHHMN6XL4ERT64FZSIMF3QZGMP4EHQERY73L3IQYJ3VCA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976656726 (526h56m57.847739761s) [noise=true , address=77.90.31.14:4662 , piece_id=HEKU46BZ64NKRGS4GTHNCVYX6KWJOMSR6J7JWUH3WZCOKGFZR3ZQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976990611 (526h56m57.848073656s) [noise=true , address=45.91.116.12:4228 , piece_id=J2EDKOABXIKKVSGZ7E3KFBOMFRXMWAFDKDUZJRALYGQYSHGCH6RQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.977447332 (526h56m57.848530367s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.977774625 (526h56m57.84885766s) [noise=true , address=45.91.116.9:4261 , piece_id=2E7LQB2AZOISLMBMICILM2H6OXC62AFTASTIJIHSU6VL4GKUR7WA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.979036097 (526h56m57.850119132s) [noise=true , address=45.91.116.10:4082 , piece_id=ESRYSZW7HJXGUQO5MEHDGTGEXLUQXMF7CKJPTATSA3AVTIEJCGJA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.981393726 (526h56m57.852476761s) [noise=true , address=145.79.227.15:4874 , piece_id=GGAUHHT6MZ76YRJ4EF4HTVMYX557SMXOL3IZJGWU7MD2I3JJ3ZTA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.982686808 (526h56m57.853769853s) [noise=true , address=145.79.227.16:4909 , piece_id=JRCQWLEITVU2FI6DPIB6RTXDXPRMLJGQEMQYSCQWS5CIWGS2D6QQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.983994349 (526h56m57.855077384s) [noise=true , address=145.79.227.10:4402 , piece_id=VIHRGMNXGF6WCUXSS2SLKH43AO73RY5WO2XNECPEN42FAOBR5Z7A, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.985345180 (526h56m57.856428215s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.986387134 (526h56m57.857470179s) [noise=true , address=45.91.116.13:4155 , piece_id=5BVUD2I2A37VTIYGCN3MUCS33ARUT6EFXVI7H4BBZKKDICI3NMLQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.986623464 (526h56m57.857706499s) [noise=true , address=77.90.31.12:4590 , piece_id=GDCIO46UXVYY2QORA5SP5ILCYB6YJ444HFOKRIGEBNGQC3FJ5FBQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:59.697992155 (526h56m59.56907519s) [upload=462204]: Next returned nil segment
Dec 27 14:25:59.698004909 (526h56m59.569087944s) [upload=462204]: Waiting for error group managing segments...
Dec 27 14:25:59.979483873 (526h56m59.850566908s) [noise=true , address=145.79.227.16:4909 , piece_id=JRCQWLEITVU2FI6DPIB6RTXDXPRMLJGQEMQYSCQWS5CIWGS2D6QQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.982814666 (526h56m59.853897711s) [noise=true , address=145.79.227.45:4229 , piece_id=OXZX5MDVEKEDFENXXNRHIPQPVOFNUPYD5QF4YIRU3MYTJMK2PCSQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.984622959 (526h56m59.855706004s) [noise=true , address=145.79.227.19:4078 , piece_id=LQS2DVYUHHMN6XL4ERT64FZSIMF3QZGMP4EHQERY73L3IQYJ3VCA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.985208203 (526h56m59.856291238s) [noise=true , address=145.79.227.10:4402 , piece_id=VIHRGMNXGF6WCUXSS2SLKH43AO73RY5WO2XNECPEN42FAOBR5Z7A, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.005223195 (526h56m59.87630623s) [noise=true , address=145.79.227.37:4565 , piece_id=4V3FY7ZJ33DZEVH27DAEDKTX64SGXT5XOFVXIIL4MH72SPHHJGWQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.010871040 (526h56m59.881954075s) [noise=true , address=145.79.227.15:4874 , piece_id=GGAUHHT6MZ76YRJ4EF4HTVMYX557SMXOL3IZJGWU7MD2I3JJ3ZTA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.237504679 (526h57m0.108587724s) [noise=true , address=45.91.116.13:4155 , piece_id=5BVUD2I2A37VTIYGCN3MUCS33ARUT6EFXVI7H4BBZKKDICI3NMLQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.273392205 (526h57m0.14447524s) [noise=true , address=77.90.31.21:4878 , piece_id=44G3B3HDGBXWV6GDDTQQ62DV6QZOPZD6DCXFL6YKITUN2URCJPHQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.340268639 (526h57m0.211351674s) [noise=true , address=45.91.116.9:4261 , piece_id=2E7LQB2AZOISLMBMICILM2H6OXC62AFTASTIJIHSU6VL4GKUR7WA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.343730832 (526h57m0.214813867s) [noise=true , address=45.91.116.10:4082 , piece_id=ESRYSZW7HJXGUQO5MEHDGTGEXLUQXMF7CKJPTATSA3AVTIEJCGJA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.820785406 (526h57m0.691868451s) [noise=true , address=77.90.31.12:4590 , piece_id=GDCIO46UXVYY2QORA5SP5ILCYB6YJ444HFOKRIGEBNGQC3FJ5FBQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.860474380 (526h57m0.731557415s) [noise=true , address=77.90.31.14:4662 , piece_id=HEKU46BZ64NKRGS4GTHNCVYX6KWJOMSR6J7JWUH3WZCOKGFZR3ZQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:01.004516601 (526h57m0.875599647s) [noise=true , address=45.91.116.12:4228 , piece_id=J2EDKOABXIKKVSGZ7E3KFBOMFRXMWAFDKDUZJRALYGQYSHGCH6RQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:01.004541869 (526h57m0.875624914s) [seg_pos={0 0}, upload=462204]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:26:01.004764193 (526h57m0.875847228s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.004772679 (526h57m0.875855714s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.004918436 (526h57m0.876001481s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.004925420 (526h57m0.876008455s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.005031813 (526h57m0.876114848s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.005048103 (526h57m0.876131148s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.005079343 (526h57m0.876162388s) [seg_pos={0 0}, upload=462204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:26:01.005175646 (526h57m0.876258682s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:26:01.005188691 (526h57m0.876271726s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:26:01.291045352 (526h57m1.162128417s) [upload=462204]: Done uploading segments...
Dec 27 14:26:01.291068015 (526h57m1.16215108s) [upload=462204]: Upload finished. err: <nil>
Duration: 5.645902169s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XX] duration:
5.000554244s trace:
https://jaeger.dev.storj.io/trace/306d4228d625194e
FAIL (2025-12-27 14:31:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000565224s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
8.107698552s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:31:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 8.107730793s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X.XXXX..X..X.....X.XXX..X...XX.............X..X....X.......] duration:
5.676814249s trace:
https://jaeger.dev.storj.io/trace/5ae325eb6324f9df
FAIL (2025-12-27 14:31:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:25:57.726806158 (526h56m57.597889203s) [upload=462204]: Starting upload
Dec 27 14:25:57.726903824 (526h56m57.597986869s) [upload=462204]: Done starting upload
Dec 27 14:25:57.726980620 (526h56m57.598063685s) [upload=462204]: Uploading segments...
Dec 27 14:25:57.727243751 (526h56m57.598326806s) [upload=462204]: Got next segment. Inline: false
Dec 27 14:25:57.727291442 (526h56m57.598374477s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:25:57.727297493 (526h56m57.598380539s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:25:57.971809967 (526h56m57.842893002s) [seg_pos={0 0}, upload=462204]: Begin upload segment...
Dec 27 14:25:57.971842348 (526h56m57.842925383s) [seg_pos={0 0}, upload=462204]: Done begin upload segment.
Dec 27 14:25:57.971953310 (526h56m57.843036345s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.973294103 (526h56m57.844377138s) [noise=true , address=77.90.31.21:4878 , piece_id=44G3B3HDGBXWV6GDDTQQ62DV6QZOPZD6DCXFL6YKITUN2URCJPHQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.975245218 (526h56m57.846328283s) [noise=true , address=145.79.227.37:4565 , piece_id=4V3FY7ZJ33DZEVH27DAEDKTX64SGXT5XOFVXIIL4MH72SPHHJGWQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.975831965 (526h56m57.84691501s) [noise=true , address=145.79.227.45:4229 , piece_id=OXZX5MDVEKEDFENXXNRHIPQPVOFNUPYD5QF4YIRU3MYTJMK2PCSQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976249150 (526h56m57.847332185s) [noise=true , address=145.79.227.19:4078 , piece_id=LQS2DVYUHHMN6XL4ERT64FZSIMF3QZGMP4EHQERY73L3IQYJ3VCA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976656726 (526h56m57.847739761s) [noise=true , address=77.90.31.14:4662 , piece_id=HEKU46BZ64NKRGS4GTHNCVYX6KWJOMSR6J7JWUH3WZCOKGFZR3ZQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.976990611 (526h56m57.848073656s) [noise=true , address=45.91.116.12:4228 , piece_id=J2EDKOABXIKKVSGZ7E3KFBOMFRXMWAFDKDUZJRALYGQYSHGCH6RQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.977447332 (526h56m57.848530367s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.977774625 (526h56m57.84885766s) [noise=true , address=45.91.116.9:4261 , piece_id=2E7LQB2AZOISLMBMICILM2H6OXC62AFTASTIJIHSU6VL4GKUR7WA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.979036097 (526h56m57.850119132s) [noise=true , address=45.91.116.10:4082 , piece_id=ESRYSZW7HJXGUQO5MEHDGTGEXLUQXMF7CKJPTATSA3AVTIEJCGJA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.981393726 (526h56m57.852476761s) [noise=true , address=145.79.227.15:4874 , piece_id=GGAUHHT6MZ76YRJ4EF4HTVMYX557SMXOL3IZJGWU7MD2I3JJ3ZTA, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.982686808 (526h56m57.853769853s) [noise=true , address=145.79.227.16:4909 , piece_id=JRCQWLEITVU2FI6DPIB6RTXDXPRMLJGQEMQYSCQWS5CIWGS2D6QQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.983994349 (526h56m57.855077384s) [noise=true , address=145.79.227.10:4402 , piece_id=VIHRGMNXGF6WCUXSS2SLKH43AO73RY5WO2XNECPEN42FAOBR5Z7A, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.985345180 (526h56m57.856428215s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.986387134 (526h56m57.857470179s) [noise=true , address=45.91.116.13:4155 , piece_id=5BVUD2I2A37VTIYGCN3MUCS33ARUT6EFXVI7H4BBZKKDICI3NMLQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:57.986623464 (526h56m57.857706499s) [noise=true , address=77.90.31.12:4590 , piece_id=GDCIO46UXVYY2QORA5SP5ILCYB6YJ444HFOKRIGEBNGQC3FJ5FBQ, seg_pos={0 0}, upload=462204]: Uploading piece...
Dec 27 14:25:59.697992155 (526h56m59.56907519s) [upload=462204]: Next returned nil segment
Dec 27 14:25:59.698004909 (526h56m59.569087944s) [upload=462204]: Waiting for error group managing segments...
Dec 27 14:25:59.979483873 (526h56m59.850566908s) [noise=true , address=145.79.227.16:4909 , piece_id=JRCQWLEITVU2FI6DPIB6RTXDXPRMLJGQEMQYSCQWS5CIWGS2D6QQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.982814666 (526h56m59.853897711s) [noise=true , address=145.79.227.45:4229 , piece_id=OXZX5MDVEKEDFENXXNRHIPQPVOFNUPYD5QF4YIRU3MYTJMK2PCSQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.984622959 (526h56m59.855706004s) [noise=true , address=145.79.227.19:4078 , piece_id=LQS2DVYUHHMN6XL4ERT64FZSIMF3QZGMP4EHQERY73L3IQYJ3VCA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:25:59.985208203 (526h56m59.856291238s) [noise=true , address=145.79.227.10:4402 , piece_id=VIHRGMNXGF6WCUXSS2SLKH43AO73RY5WO2XNECPEN42FAOBR5Z7A, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.005223195 (526h56m59.87630623s) [noise=true , address=145.79.227.37:4565 , piece_id=4V3FY7ZJ33DZEVH27DAEDKTX64SGXT5XOFVXIIL4MH72SPHHJGWQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.010871040 (526h56m59.881954075s) [noise=true , address=145.79.227.15:4874 , piece_id=GGAUHHT6MZ76YRJ4EF4HTVMYX557SMXOL3IZJGWU7MD2I3JJ3ZTA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.237504679 (526h57m0.108587724s) [noise=true , address=45.91.116.13:4155 , piece_id=5BVUD2I2A37VTIYGCN3MUCS33ARUT6EFXVI7H4BBZKKDICI3NMLQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.273392205 (526h57m0.14447524s) [noise=true , address=77.90.31.21:4878 , piece_id=44G3B3HDGBXWV6GDDTQQ62DV6QZOPZD6DCXFL6YKITUN2URCJPHQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.340268639 (526h57m0.211351674s) [noise=true , address=45.91.116.9:4261 , piece_id=2E7LQB2AZOISLMBMICILM2H6OXC62AFTASTIJIHSU6VL4GKUR7WA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.343730832 (526h57m0.214813867s) [noise=true , address=45.91.116.10:4082 , piece_id=ESRYSZW7HJXGUQO5MEHDGTGEXLUQXMF7CKJPTATSA3AVTIEJCGJA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.820785406 (526h57m0.691868451s) [noise=true , address=77.90.31.12:4590 , piece_id=GDCIO46UXVYY2QORA5SP5ILCYB6YJ444HFOKRIGEBNGQC3FJ5FBQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:00.860474380 (526h57m0.731557415s) [noise=true , address=77.90.31.14:4662 , piece_id=HEKU46BZ64NKRGS4GTHNCVYX6KWJOMSR6J7JWUH3WZCOKGFZR3ZQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:01.004516601 (526h57m0.875599647s) [noise=true , address=45.91.116.12:4228 , piece_id=J2EDKOABXIKKVSGZ7E3KFBOMFRXMWAFDKDUZJRALYGQYSHGCH6RQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: <nil>
Dec 27 14:26:01.004541869 (526h57m0.875624914s) [seg_pos={0 0}, upload=462204]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:26:01.004764193 (526h57m0.875847228s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.004772679 (526h57m0.875855714s) [noise=true , address=77.90.31.10:4518 , piece_id=XC3SYAXA7YCYVWMM2LSUFHU6FGCS57NFMIDQ5LMLJO4SOGCEJ53Q, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.004918436 (526h57m0.876001481s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.004925420 (526h57m0.876008455s) [noise=true , address=45.91.116.7:4009 , piece_id=ZZ2MENTHYOYBNPCT2UF4VSRP4NUX7SNDQ23UQ7GNF5BURRDB25JA, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.005031813 (526h57m0.876114848s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 27 14:26:01.005048103 (526h57m0.876131148s) [noise=true , address=145.79.227.33:4871 , piece_id=5YIJINPXBOAWCL7ML57QWSBRDTQTFXHJP4O4AYH7KIIEH5E3I7HQ, seg_pos={0 0}, upload=462204]: Long tail cancellation due to optimal threshold
Dec 27 14:26:01.005079343 (526h57m0.876162388s) [seg_pos={0 0}, upload=462204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:26:01.005175646 (526h57m0.876258682s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:26:01.005188691 (526h57m0.876271726s) [upload=462204]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:26:01.291045352 (526h57m1.162128417s) [upload=462204]: Done uploading segments...
Dec 27 14:26:01.291068015 (526h57m1.16215108s) [upload=462204]: Upload finished. err: <nil>
Duration: 5.676842101s
core
uplink
us-select-1
|