Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......XXX...] duration:
3.428005821s trace:
https://jaeger.dev.storj.io/trace/480d8552d5115fd1
FAIL (2026-01-07 19:47:44 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.428038393s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X..X.X.X....X........X..........X..XX..X.X....XX......X.X.] duration:
6.05438692s trace:
https://jaeger.dev.storj.io/trace/406c98c200d07aff
FAIL (2026-01-07 19:47:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HCSOI4JTZH6DKXSXYACLGWXL7JN2C63FYMIMIRJMLX5NWAN6C7TQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LARZGT662LJNOGHK7CU7AFJM3FIAJSYFSEBGXVGK3DPKSTN57UYQ) context deadline exceeded
======LOGS======
Jan 7 19:46:30.808653806 (796h17m30.679736851s) [upload=696461]: Starting upload
Jan 7 19:46:30.808744198 (796h17m30.679827233s) [upload=696461]: Done starting upload
Jan 7 19:46:30.808801817 (796h17m30.679884852s) [upload=696461]: Uploading segments...
Jan 7 19:46:30.809157915 (796h17m30.68024095s) [upload=696461]: Got next segment. Inline: false
Jan 7 19:46:30.809211828 (796h17m30.680294863s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:46:30.809218040 (796h17m30.680301085s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:46:31.055240255 (796h17m30.92632329s) [seg_pos={0 0}, upload=696461]: Begin upload segment...
Jan 7 19:46:31.055303055 (796h17m30.92638609s) [seg_pos={0 0}, upload=696461]: Done begin upload segment.
Jan 7 19:46:31.055339164 (796h17m30.926422199s) [noise=true , address=45.91.116.4:4939 , piece_id=6RQ5U27OXZY7YMZT3GFQLRTK4YQPB7NYSXVI7DCHIPN5TRLMHXPA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.055388718 (796h17m30.926471783s) [noise=true , address=77.90.31.14:4663 , piece_id=LARZGT662LJNOGHK7CU7AFJM3FIAJSYFSEBGXVGK3DPKSTN57UYQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.055856728 (796h17m30.926939764s) [noise=true , address=145.79.227.7:4294 , piece_id=73U64TWLCK7Z33EJCWX6IQ5E4L2J7RWORPP24VEGXYA2DJFGEQVQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.056152142 (796h17m30.927235177s) [noise=true , address=145.79.227.29:4943 , piece_id=CZNB3V65NP7RMUW2VSSHTLKPTQLSNW7KSX74SNMVVGHWGMLZDM3Q, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.056523910 (796h17m30.927606955s) [noise=true , address=145.79.227.42:4122 , piece_id=2OUSYYJVVZQDFWLNVGO62R2ISCGE5NICMQPVL7PPBOSJHPVVMSDQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.056896098 (796h17m30.927979133s) [noise=true , address=145.79.227.31:4799 , piece_id=OIDKGEOTBHSMIGQFER6GZNBCVQ3ALABFJWHNOP3GKNYQMIDLIHVQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.057216377 (796h17m30.928299412s) [noise=true , address=145.79.227.16:4911 , piece_id=MDJTZGLGHZI665MSFBPWOZZBBH35AOXWZGIEIUKJE34ZMYYUSUFA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.057645746 (796h17m30.928728781s) [noise=true , address=145.79.227.48:4302 , piece_id=HZKQPYG6KQLBRYGMGXADOMJW3OSFX33KTQWESFU26EGPP3OOPI5Q, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058076826 (796h17m30.929159871s) [noise=true , address=145.79.227.52:4446 , piece_id=72AI5O7DPURA44KKWXOL5D6YMFXAR7FKHQND7UHA25A6YR7BNZRA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058086354 (796h17m30.929169389s) [noise=true , address=145.79.227.34:4907 , piece_id=Q3R2A2MLFZJ7VHBI32KK63THF4OC5MRDTUTGGPBN3D2MJ7W3GXFQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058453163 (796h17m30.929536198s) [noise=true , address=145.79.227.2:4114 , piece_id=6IZPR2DB5AV3FF77SBQQXJ4WQEZKTCFPQS2XKDRTLNJYOSMAUYIQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058484573 (796h17m30.929567618s) [noise=true , address=45.91.116.10:4083 , piece_id=HCSOI4JTZH6DKXSXYACLGWXL7JN2C63FYMIMIRJMLX5NWAN6C7TQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058817085 (796h17m30.929900121s) [noise=true , address=145.79.227.18:4006 , piece_id=ZG33UJTQVNDYCIBNZOIYGPO2ZOEN2Z5ENN7TRIUJXXRR5Y6NVJZQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.059182462 (796h17m30.930265497s) [noise=true , address=145.79.227.6:4259 , piece_id=GBZTPNJVFGBKDDVQE6TNH6DGR3X5L3BXLRTCKPOU3KCWU75FUHPQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.059415976 (796h17m30.930499011s) [noise=true , address=145.79.227.15:4874 , piece_id=LTIFJBSMXQNUL7VHS6UWIPBNLGEOHOTG2WVFTE453JQJVCTYD2OA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.059529773 (796h17m30.930612808s) [noise=true , address=145.79.227.21:4473 , piece_id=6K5Y4KY3RWRVVKPJEDHJI5HOB4GGAAZTQAG3HZY2ZAX4JIHNW35A, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:32.676601230 (796h17m32.547684265s) [upload=696461]: Next returned nil segment
Jan 7 19:46:32.676617521 (796h17m32.547700566s) [upload=696461]: Waiting for error group managing segments...
Jan 7 19:46:33.033160715 (796h17m32.90424376s) [noise=true , address=145.79.227.2:4114 , piece_id=6IZPR2DB5AV3FF77SBQQXJ4WQEZKTCFPQS2XKDRTLNJYOSMAUYIQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.036195634 (796h17m32.907278679s) [noise=true , address=145.79.227.48:4302 , piece_id=HZKQPYG6KQLBRYGMGXADOMJW3OSFX33KTQWESFU26EGPP3OOPI5Q, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.054685812 (796h17m32.925768847s) [noise=true , address=145.79.227.52:4446 , piece_id=72AI5O7DPURA44KKWXOL5D6YMFXAR7FKHQND7UHA25A6YR7BNZRA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.062624549 (796h17m32.933707584s) [noise=true , address=145.79.227.6:4259 , piece_id=GBZTPNJVFGBKDDVQE6TNH6DGR3X5L3BXLRTCKPOU3KCWU75FUHPQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.063988346 (796h17m32.935071391s) [noise=true , address=145.79.227.42:4122 , piece_id=2OUSYYJVVZQDFWLNVGO62R2ISCGE5NICMQPVL7PPBOSJHPVVMSDQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.080448409 (796h17m32.951531444s) [noise=true , address=145.79.227.34:4907 , piece_id=Q3R2A2MLFZJ7VHBI32KK63THF4OC5MRDTUTGGPBN3D2MJ7W3GXFQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.089497579 (796h17m32.960580624s) [noise=true , address=145.79.227.7:4294 , piece_id=73U64TWLCK7Z33EJCWX6IQ5E4L2J7RWORPP24VEGXYA2DJFGEQVQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.113718009 (796h17m32.984801044s) [noise=true , address=145.79.227.18:4006 , piece_id=ZG33UJTQVNDYCIBNZOIYGPO2ZOEN2Z5ENN7TRIUJXXRR5Y6NVJZQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.140873637 (796h17m33.011956672s) [noise=true , address=145.79.227.15:4874 , piece_id=LTIFJBSMXQNUL7VHS6UWIPBNLGEOHOTG2WVFTE453JQJVCTYD2OA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.204129967 (796h17m33.075213012s) [noise=true , address=145.79.227.16:4911 , piece_id=MDJTZGLGHZI665MSFBPWOZZBBH35AOXWZGIEIUKJE34ZMYYUSUFA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.349819372 (796h17m33.220902417s) [noise=true , address=77.90.31.14:4663 , piece_id=LARZGT662LJNOGHK7CU7AFJM3FIAJSYFSEBGXVGK3DPKSTN57UYQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.468808516 (796h17m33.339891561s) [noise=true , address=45.91.116.10:4083 , piece_id=HCSOI4JTZH6DKXSXYACLGWXL7JN2C63FYMIMIRJMLX5NWAN6C7TQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.806985960 (796h17m33.678068995s) [noise=true , address=45.91.116.4:4939 , piece_id=6RQ5U27OXZY7YMZT3GFQLRTK4YQPB7NYSXVI7DCHIPN5TRLMHXPA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.807023070 (796h17m33.678106105s) [seg_pos={0 0}, upload=696461]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:46:33.807200638 (796h17m33.678283683s) [noise=true , address=145.79.227.31:4799 , piece_id=OIDKGEOTBHSMIGQFER6GZNBCVQ3ALABFJWHNOP3GKNYQMIDLIHVQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF
Jan 7 19:46:33.807206329 (796h17m33.678289364s) [noise=true , address=145.79.227.31:4799 , piece_id=OIDKGEOTBHSMIGQFER6GZNBCVQ3ALABFJWHNOP3GKNYQMIDLIHVQ, seg_pos={0 0}, upload=696461]: Long tail cancellation due to optimal threshold
Jan 7 19:46:33.807417290 (796h17m33.678500326s) [noise=true , address=145.79.227.29:4943 , piece_id=CZNB3V65NP7RMUW2VSSHTLKPTQLSNW7KSX74SNMVVGHWGMLZDM3Q, seg_pos={0 0}, upload=696461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Jan 7 19:46:33.807423072 (796h17m33.678506107s) [noise=true , address=145.79.227.29:4943 , piece_id=CZNB3V65NP7RMUW2VSSHTLKPTQLSNW7KSX74SNMVVGHWGMLZDM3Q, seg_pos={0 0}, upload=696461]: Long tail cancellation due to optimal threshold
Jan 7 19:46:33.807426549 (796h17m33.678509604s) [noise=true , address=145.79.227.21:4473 , piece_id=6K5Y4KY3RWRVVKPJEDHJI5HOB4GGAAZTQAG3HZY2ZAX4JIHNW35A, seg_pos={0 0}, upload=696461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:46:33.807448300 (796h17m33.678531345s) [noise=true , address=145.79.227.21:4473 , piece_id=6K5Y4KY3RWRVVKPJEDHJI5HOB4GGAAZTQAG3HZY2ZAX4JIHNW35A, seg_pos={0 0}, upload=696461]: Long tail cancellation due to optimal threshold
Jan 7 19:46:33.807472356 (796h17m33.678555391s) [seg_pos={0 0}, upload=696461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:46:33.807538161 (796h17m33.678621196s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:46:33.807543822 (796h17m33.678626857s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:46:34.105326456 (796h17m33.976409501s) [upload=696461]: Done uploading segments...
Jan 7 19:46:34.105348558 (796h17m33.976431604s) [upload=696461]: Upload finished. err: <nil>
Duration: 6.054413852s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
274.804761ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae
FAIL (2026-01-07 19:47:36 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 274.838125ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
1.117570897s remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/521974c405507e8c
FAIL (2026-01-07 19:47:35 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 1.117600073s
edge
linksharing
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
27 out of 60 is failed (threshold=6) [X..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......XXX..] duration:
3.712549204s trace:
https://jaeger.dev.storj.io/trace/480d8552d5115fd1
FAIL (2026-01-07 19:47:25 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.712576266s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X..X..X.X.X....X........X..........X..XX..X.X....XX......X.X] duration:
30.003580903s trace:
https://jaeger.dev.storj.io/trace/406c98c200d07aff
FAIL (2026-01-07 19:47:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HCSOI4JTZH6DKXSXYACLGWXL7JN2C63FYMIMIRJMLX5NWAN6C7TQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LARZGT662LJNOGHK7CU7AFJM3FIAJSYFSEBGXVGK3DPKSTN57UYQ) context deadline exceeded
======LOGS======
Jan 7 19:46:30.808653806 (796h17m30.679736851s) [upload=696461]: Starting upload
Jan 7 19:46:30.808744198 (796h17m30.679827233s) [upload=696461]: Done starting upload
Jan 7 19:46:30.808801817 (796h17m30.679884852s) [upload=696461]: Uploading segments...
Jan 7 19:46:30.809157915 (796h17m30.68024095s) [upload=696461]: Got next segment. Inline: false
Jan 7 19:46:30.809211828 (796h17m30.680294863s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:46:30.809218040 (796h17m30.680301085s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:46:31.055240255 (796h17m30.92632329s) [seg_pos={0 0}, upload=696461]: Begin upload segment...
Jan 7 19:46:31.055303055 (796h17m30.92638609s) [seg_pos={0 0}, upload=696461]: Done begin upload segment.
Jan 7 19:46:31.055339164 (796h17m30.926422199s) [noise=true , address=45.91.116.4:4939 , piece_id=6RQ5U27OXZY7YMZT3GFQLRTK4YQPB7NYSXVI7DCHIPN5TRLMHXPA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.055388718 (796h17m30.926471783s) [noise=true , address=77.90.31.14:4663 , piece_id=LARZGT662LJNOGHK7CU7AFJM3FIAJSYFSEBGXVGK3DPKSTN57UYQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.055856728 (796h17m30.926939764s) [noise=true , address=145.79.227.7:4294 , piece_id=73U64TWLCK7Z33EJCWX6IQ5E4L2J7RWORPP24VEGXYA2DJFGEQVQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.056152142 (796h17m30.927235177s) [noise=true , address=145.79.227.29:4943 , piece_id=CZNB3V65NP7RMUW2VSSHTLKPTQLSNW7KSX74SNMVVGHWGMLZDM3Q, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.056523910 (796h17m30.927606955s) [noise=true , address=145.79.227.42:4122 , piece_id=2OUSYYJVVZQDFWLNVGO62R2ISCGE5NICMQPVL7PPBOSJHPVVMSDQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.056896098 (796h17m30.927979133s) [noise=true , address=145.79.227.31:4799 , piece_id=OIDKGEOTBHSMIGQFER6GZNBCVQ3ALABFJWHNOP3GKNYQMIDLIHVQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.057216377 (796h17m30.928299412s) [noise=true , address=145.79.227.16:4911 , piece_id=MDJTZGLGHZI665MSFBPWOZZBBH35AOXWZGIEIUKJE34ZMYYUSUFA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.057645746 (796h17m30.928728781s) [noise=true , address=145.79.227.48:4302 , piece_id=HZKQPYG6KQLBRYGMGXADOMJW3OSFX33KTQWESFU26EGPP3OOPI5Q, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058076826 (796h17m30.929159871s) [noise=true , address=145.79.227.52:4446 , piece_id=72AI5O7DPURA44KKWXOL5D6YMFXAR7FKHQND7UHA25A6YR7BNZRA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058086354 (796h17m30.929169389s) [noise=true , address=145.79.227.34:4907 , piece_id=Q3R2A2MLFZJ7VHBI32KK63THF4OC5MRDTUTGGPBN3D2MJ7W3GXFQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058453163 (796h17m30.929536198s) [noise=true , address=145.79.227.2:4114 , piece_id=6IZPR2DB5AV3FF77SBQQXJ4WQEZKTCFPQS2XKDRTLNJYOSMAUYIQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058484573 (796h17m30.929567618s) [noise=true , address=45.91.116.10:4083 , piece_id=HCSOI4JTZH6DKXSXYACLGWXL7JN2C63FYMIMIRJMLX5NWAN6C7TQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.058817085 (796h17m30.929900121s) [noise=true , address=145.79.227.18:4006 , piece_id=ZG33UJTQVNDYCIBNZOIYGPO2ZOEN2Z5ENN7TRIUJXXRR5Y6NVJZQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.059182462 (796h17m30.930265497s) [noise=true , address=145.79.227.6:4259 , piece_id=GBZTPNJVFGBKDDVQE6TNH6DGR3X5L3BXLRTCKPOU3KCWU75FUHPQ, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.059415976 (796h17m30.930499011s) [noise=true , address=145.79.227.15:4874 , piece_id=LTIFJBSMXQNUL7VHS6UWIPBNLGEOHOTG2WVFTE453JQJVCTYD2OA, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:31.059529773 (796h17m30.930612808s) [noise=true , address=145.79.227.21:4473 , piece_id=6K5Y4KY3RWRVVKPJEDHJI5HOB4GGAAZTQAG3HZY2ZAX4JIHNW35A, seg_pos={0 0}, upload=696461]: Uploading piece...
Jan 7 19:46:32.676601230 (796h17m32.547684265s) [upload=696461]: Next returned nil segment
Jan 7 19:46:32.676617521 (796h17m32.547700566s) [upload=696461]: Waiting for error group managing segments...
Jan 7 19:46:33.033160715 (796h17m32.90424376s) [noise=true , address=145.79.227.2:4114 , piece_id=6IZPR2DB5AV3FF77SBQQXJ4WQEZKTCFPQS2XKDRTLNJYOSMAUYIQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.036195634 (796h17m32.907278679s) [noise=true , address=145.79.227.48:4302 , piece_id=HZKQPYG6KQLBRYGMGXADOMJW3OSFX33KTQWESFU26EGPP3OOPI5Q, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.054685812 (796h17m32.925768847s) [noise=true , address=145.79.227.52:4446 , piece_id=72AI5O7DPURA44KKWXOL5D6YMFXAR7FKHQND7UHA25A6YR7BNZRA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.062624549 (796h17m32.933707584s) [noise=true , address=145.79.227.6:4259 , piece_id=GBZTPNJVFGBKDDVQE6TNH6DGR3X5L3BXLRTCKPOU3KCWU75FUHPQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.063988346 (796h17m32.935071391s) [noise=true , address=145.79.227.42:4122 , piece_id=2OUSYYJVVZQDFWLNVGO62R2ISCGE5NICMQPVL7PPBOSJHPVVMSDQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.080448409 (796h17m32.951531444s) [noise=true , address=145.79.227.34:4907 , piece_id=Q3R2A2MLFZJ7VHBI32KK63THF4OC5MRDTUTGGPBN3D2MJ7W3GXFQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.089497579 (796h17m32.960580624s) [noise=true , address=145.79.227.7:4294 , piece_id=73U64TWLCK7Z33EJCWX6IQ5E4L2J7RWORPP24VEGXYA2DJFGEQVQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.113718009 (796h17m32.984801044s) [noise=true , address=145.79.227.18:4006 , piece_id=ZG33UJTQVNDYCIBNZOIYGPO2ZOEN2Z5ENN7TRIUJXXRR5Y6NVJZQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.140873637 (796h17m33.011956672s) [noise=true , address=145.79.227.15:4874 , piece_id=LTIFJBSMXQNUL7VHS6UWIPBNLGEOHOTG2WVFTE453JQJVCTYD2OA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.204129967 (796h17m33.075213012s) [noise=true , address=145.79.227.16:4911 , piece_id=MDJTZGLGHZI665MSFBPWOZZBBH35AOXWZGIEIUKJE34ZMYYUSUFA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.349819372 (796h17m33.220902417s) [noise=true , address=77.90.31.14:4663 , piece_id=LARZGT662LJNOGHK7CU7AFJM3FIAJSYFSEBGXVGK3DPKSTN57UYQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.468808516 (796h17m33.339891561s) [noise=true , address=45.91.116.10:4083 , piece_id=HCSOI4JTZH6DKXSXYACLGWXL7JN2C63FYMIMIRJMLX5NWAN6C7TQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.806985960 (796h17m33.678068995s) [noise=true , address=45.91.116.4:4939 , piece_id=6RQ5U27OXZY7YMZT3GFQLRTK4YQPB7NYSXVI7DCHIPN5TRLMHXPA, seg_pos={0 0}, upload=696461]: Done uploading piece. err: <nil>
Jan 7 19:46:33.807023070 (796h17m33.678106105s) [seg_pos={0 0}, upload=696461]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:46:33.807200638 (796h17m33.678283683s) [noise=true , address=145.79.227.31:4799 , piece_id=OIDKGEOTBHSMIGQFER6GZNBCVQ3ALABFJWHNOP3GKNYQMIDLIHVQ, seg_pos={0 0}, upload=696461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF
Jan 7 19:46:33.807206329 (796h17m33.678289364s) [noise=true , address=145.79.227.31:4799 , piece_id=OIDKGEOTBHSMIGQFER6GZNBCVQ3ALABFJWHNOP3GKNYQMIDLIHVQ, seg_pos={0 0}, upload=696461]: Long tail cancellation due to optimal threshold
Jan 7 19:46:33.807417290 (796h17m33.678500326s) [noise=true , address=145.79.227.29:4943 , piece_id=CZNB3V65NP7RMUW2VSSHTLKPTQLSNW7KSX74SNMVVGHWGMLZDM3Q, seg_pos={0 0}, upload=696461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Jan 7 19:46:33.807423072 (796h17m33.678506107s) [noise=true , address=145.79.227.29:4943 , piece_id=CZNB3V65NP7RMUW2VSSHTLKPTQLSNW7KSX74SNMVVGHWGMLZDM3Q, seg_pos={0 0}, upload=696461]: Long tail cancellation due to optimal threshold
Jan 7 19:46:33.807426549 (796h17m33.678509604s) [noise=true , address=145.79.227.21:4473 , piece_id=6K5Y4KY3RWRVVKPJEDHJI5HOB4GGAAZTQAG3HZY2ZAX4JIHNW35A, seg_pos={0 0}, upload=696461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:46:33.807448300 (796h17m33.678531345s) [noise=true , address=145.79.227.21:4473 , piece_id=6K5Y4KY3RWRVVKPJEDHJI5HOB4GGAAZTQAG3HZY2ZAX4JIHNW35A, seg_pos={0 0}, upload=696461]: Long tail cancellation due to optimal threshold
Jan 7 19:46:33.807472356 (796h17m33.678555391s) [seg_pos={0 0}, upload=696461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:46:33.807538161 (796h17m33.678621196s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:46:33.807543822 (796h17m33.678626857s) [upload=696461]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:46:34.105326456 (796h17m33.976409501s) [upload=696461]: Done uploading segments...
Jan 7 19:46:34.105348558 (796h17m33.976431604s) [upload=696461]: Upload finished. err: <nil>
Duration: 30.003596432s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
28 out of 60 is failed (threshold=6) [XX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......XXX.] duration:
3.526651426s trace:
https://jaeger.dev.storj.io/trace/480d8552d5115fd1
FAIL (2026-01-07 19:46:57 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.526674669s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [...X......................................................X.] duration:
411.142218ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/216ac0c763a6ab13
FAIL (2026-01-07 19:46:42 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 411.159583ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
472.175136ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/521974c405507e8c
FAIL (2026-01-07 19:46:40 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 472.185816ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
2m0.001059305s remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/521974c405507e8c
FAIL (2026-01-07 19:46:39 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 2m0.001087019s
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
414.145507ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae
FAIL (2026-01-07 19:46:32 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 414.569786ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
2m0.000522361s remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/2cc1d2bf872e36ae
FAIL (2026-01-07 19:46:31 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 2m0.000546508s
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
29 out of 60 is failed (threshold=6) [XXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......XXX] duration:
5.001052627s trace:
https://jaeger.dev.storj.io/trace/480d8552d5115fd1
FAIL (2026-01-07 19:46:19 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.001081441s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
29 out of 60 is failed (threshold=6) [XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......XX] duration:
5.000511097s trace:
https://jaeger.dev.storj.io/trace/318bea41f6868d86
FAIL (2026-01-07 19:46:02 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000538118s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X..X..X.X.X....X........X..........X..XX..X.X....XX......X.] duration:
5.821930327s trace:
https://jaeger.dev.storj.io/trace/4557ed9b8e31de9c
FAIL (2026-01-07 19:45:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: N63NSYUX7FOAQOIGHR7ZAOJT54WROCD46WUAK5V5KM6MBTXMLIDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YAOAYOGN6IMTMQJFLFGZXFQJPOS5T5QPOJPQDUH5N2R4GGBY45AQ) context deadline exceeded
======LOGS======
Jan 7 19:44:08.332765844 (796h15m8.203848889s) [upload=696432]: Starting upload
Jan 7 19:44:08.332974742 (796h15m8.204057807s) [upload=696432]: Done starting upload
Jan 7 19:44:08.333217003 (796h15m8.204300038s) [upload=696432]: Uploading segments...
Jan 7 19:44:08.333336681 (796h15m8.204419736s) [upload=696432]: Got next segment. Inline: false
Jan 7 19:44:08.333377820 (796h15m8.204460855s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:44:08.333384442 (796h15m8.204467487s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:44:08.588937856 (796h15m8.460020901s) [seg_pos={0 0}, upload=696432]: Begin upload segment...
Jan 7 19:44:08.588994554 (796h15m8.460077589s) [seg_pos={0 0}, upload=696432]: Done begin upload segment.
Jan 7 19:44:08.589025764 (796h15m8.460108809s) [noise=true , address=45.91.116.7:4012 , piece_id=YAOAYOGN6IMTMQJFLFGZXFQJPOS5T5QPOJPQDUH5N2R4GGBY45AQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.589087351 (796h15m8.460170396s) [noise=true , address=145.79.227.15:4873 , piece_id=NIXB5RF73OY52CVGPNHG4X7GYVHZHMSJ5UPGBN2QT2NIJBNEA6QQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.589863268 (796h15m8.460946303s) [noise=true , address=145.79.227.20:4438 , piece_id=45WLNZXX7J7FNUAWTHWZSCVPVZ6ZVTQHPU5XLWGEMXN2ITLAYI7Q, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.590381135 (796h15m8.46146418s) [noise=true , address=145.79.227.35:4482 , piece_id=DCTTZGQIEMP5P2KGRKHXU3SL2LTU4PROWDHP33LT2MLBJZDYVC5Q, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.590896996 (796h15m8.461980041s) [noise=true , address=45.91.116.14:4190 , piece_id=TP2NJXIJXVUKSGLHXI34MFK57UEZZ253EFSAY3QCAJYTE36L3ZRQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.591341222 (796h15m8.462424267s) [noise=true , address=145.79.227.11:4730 , piece_id=4VVGNGW4CKDPBUPKMAHEBASKY2HZ2SEF25C6JUUDTVFU2L3QVSFQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.592060773 (796h15m8.463143818s) [noise=true , address=145.79.227.52:4445 , piece_id=JXL6ZIUQBVVI62FDOQL4WPX4NWCG5YHKRCKDNPLNAG4YWVFXMWZQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.592560794 (796h15m8.46364384s) [noise=true , address=145.79.227.34:4906 , piece_id=2CF2WFERVCOIXVV636BNWNOFTRQ6LRL54UOTRKDZCXBDHDRHASQA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.593282850 (796h15m8.464365895s) [noise=true , address=145.79.227.13:4804 , piece_id=STTU2CXMNNA2FN4JDUNU7T6LWXPICSQUS4G5GZNY7RQ6W4DC67XA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.594933733 (796h15m8.466016778s) [noise=true , address=45.91.116.11:4118 , piece_id=N63NSYUX7FOAQOIGHR7ZAOJT54WROCD46WUAK5V5KM6MBTXMLIDQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.595392266 (796h15m8.466475301s) [noise=true , address=45.91.116.5:4297 , piece_id=VRDCLQSPZDUJB3YQINMQDCYZCASXDKF4MRA2SFXUGXRXGC7UE2YQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.595852353 (796h15m8.466935388s) [noise=true , address=145.79.227.29:4941 , piece_id=EUMSEXARZZLLKIFMU7KNR572KOJY6JKJ4CEXILOZG7EL45HWAZEA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.596283133 (796h15m8.467366178s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2M7NQQMSKVZMFLT4DPIGAJ5V5V3OZV5ENJB7OD55RV7OOWFYBCQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.596691981 (796h15m8.467775026s) [noise=true , address=145.79.227.41:4087 , piece_id=XLT5LPCCTO5YQMXRJVWZ72JQIUNJOZYHBIKC5VFFANKIICBTYFVQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.597184539 (796h15m8.468267584s) [noise=true , address=145.79.227.47:4265 , piece_id=B5PC3HASGN5SU7KBRRPSFINAASBTS5GSYCT5ZYEYP2FGQOHPVXBQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.597673530 (796h15m8.468756575s) [noise=true , address=45.91.116.4:4939 , piece_id=C4CX7XDCFMA5CKPWQAUHJHCSPL24MUKPCIMDBGJF6GJ5W6JNHEQA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:10.202592207 (796h15m10.073675252s) [upload=696432]: Next returned nil segment
Jan 7 19:44:10.202638124 (796h15m10.073721169s) [upload=696432]: Waiting for error group managing segments...
Jan 7 19:44:10.586509756 (796h15m10.457592801s) [noise=true , address=145.79.227.11:4730 , piece_id=4VVGNGW4CKDPBUPKMAHEBASKY2HZ2SEF25C6JUUDTVFU2L3QVSFQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.589115648 (796h15m10.460198693s) [noise=true , address=145.79.227.15:4873 , piece_id=NIXB5RF73OY52CVGPNHG4X7GYVHZHMSJ5UPGBN2QT2NIJBNEA6QQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.589194348 (796h15m10.460277393s) [noise=true , address=145.79.227.35:4482 , piece_id=DCTTZGQIEMP5P2KGRKHXU3SL2LTU4PROWDHP33LT2MLBJZDYVC5Q, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.594279190 (796h15m10.465362235s) [noise=true , address=145.79.227.47:4265 , piece_id=B5PC3HASGN5SU7KBRRPSFINAASBTS5GSYCT5ZYEYP2FGQOHPVXBQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.597989425 (796h15m10.46907247s) [noise=true , address=145.79.227.20:4438 , piece_id=45WLNZXX7J7FNUAWTHWZSCVPVZ6ZVTQHPU5XLWGEMXN2ITLAYI7Q, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.601834046 (796h15m10.472917081s) [noise=true , address=145.79.227.52:4445 , piece_id=JXL6ZIUQBVVI62FDOQL4WPX4NWCG5YHKRCKDNPLNAG4YWVFXMWZQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.603204615 (796h15m10.47428766s) [noise=true , address=145.79.227.41:4087 , piece_id=XLT5LPCCTO5YQMXRJVWZ72JQIUNJOZYHBIKC5VFFANKIICBTYFVQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.607187650 (796h15m10.478270695s) [noise=true , address=145.79.227.34:4906 , piece_id=2CF2WFERVCOIXVV636BNWNOFTRQ6LRL54UOTRKDZCXBDHDRHASQA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.658088834 (796h15m10.529171869s) [noise=true , address=145.79.227.13:4804 , piece_id=STTU2CXMNNA2FN4JDUNU7T6LWXPICSQUS4G5GZNY7RQ6W4DC67XA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.987236272 (796h15m10.858319307s) [noise=true , address=45.91.116.11:4118 , piece_id=N63NSYUX7FOAQOIGHR7ZAOJT54WROCD46WUAK5V5KM6MBTXMLIDQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.007251394 (796h15m10.878334459s) [noise=true , address=45.91.116.14:4190 , piece_id=TP2NJXIJXVUKSGLHXI34MFK57UEZZ253EFSAY3QCAJYTE36L3ZRQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.152638965 (796h15m11.02372201s) [noise=true , address=45.91.116.7:4012 , piece_id=YAOAYOGN6IMTMQJFLFGZXFQJPOS5T5QPOJPQDUH5N2R4GGBY45AQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.195362016 (796h15m11.066445061s) [noise=true , address=45.91.116.4:4939 , piece_id=C4CX7XDCFMA5CKPWQAUHJHCSPL24MUKPCIMDBGJF6GJ5W6JNHEQA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.195383697 (796h15m11.066466732s) [seg_pos={0 0}, upload=696432]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:44:11.195634255 (796h15m11.0667173s) [noise=true , address=145.79.227.29:4941 , piece_id=EUMSEXARZZLLKIFMU7KNR572KOJY6JKJ4CEXILOZG7EL45HWAZEA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Jan 7 19:44:11.195642560 (796h15m11.066725606s) [noise=true , address=145.79.227.29:4941 , piece_id=EUMSEXARZZLLKIFMU7KNR572KOJY6JKJ4CEXILOZG7EL45HWAZEA, seg_pos={0 0}, upload=696432]: Long tail cancellation due to optimal threshold
Jan 7 19:44:11.195899650 (796h15m11.066982685s) [noise=true , address=45.91.116.5:4297 , piece_id=VRDCLQSPZDUJB3YQINMQDCYZCASXDKF4MRA2SFXUGXRXGC7UE2YQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Jan 7 19:44:11.195909068 (796h15m11.066992103s) [noise=true , address=45.91.116.5:4297 , piece_id=VRDCLQSPZDUJB3YQINMQDCYZCASXDKF4MRA2SFXUGXRXGC7UE2YQ, seg_pos={0 0}, upload=696432]: Long tail cancellation due to optimal threshold
Jan 7 19:44:11.195924257 (796h15m11.067007312s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2M7NQQMSKVZMFLT4DPIGAJ5V5V3OZV5ENJB7OD55RV7OOWFYBCQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Jan 7 19:44:11.195945958 (796h15m11.067029004s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2M7NQQMSKVZMFLT4DPIGAJ5V5V3OZV5ENJB7OD55RV7OOWFYBCQ, seg_pos={0 0}, upload=696432]: Long tail cancellation due to optimal threshold
Jan 7 19:44:11.195980685 (796h15m11.06706373s) [seg_pos={0 0}, upload=696432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:44:11.196077739 (796h15m11.067160784s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:44:11.196084372 (796h15m11.067167417s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:44:11.480318939 (796h15m11.351401984s) [upload=696432]: Done uploading segments...
Jan 7 19:44:11.480349377 (796h15m11.351432412s) [upload=696432]: Upload finished. err: <nil>
Duration: 5.821950365s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
28 out of 60 is failed (threshold=6) [.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......X] duration:
5.001371463s trace:
https://jaeger.dev.storj.io/trace/287b6b2d93a84943
FAIL (2026-01-07 19:44:54 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.00139554s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X..X..X.X.X....X........X..........X..XX..X.X....XX......X] duration:
30.002487579s trace:
https://jaeger.dev.storj.io/trace/4557ed9b8e31de9c
FAIL (2026-01-07 19:44:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: N63NSYUX7FOAQOIGHR7ZAOJT54WROCD46WUAK5V5KM6MBTXMLIDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YAOAYOGN6IMTMQJFLFGZXFQJPOS5T5QPOJPQDUH5N2R4GGBY45AQ) context deadline exceeded
======LOGS======
Jan 7 19:44:08.332765844 (796h15m8.203848889s) [upload=696432]: Starting upload
Jan 7 19:44:08.332974742 (796h15m8.204057807s) [upload=696432]: Done starting upload
Jan 7 19:44:08.333217003 (796h15m8.204300038s) [upload=696432]: Uploading segments...
Jan 7 19:44:08.333336681 (796h15m8.204419736s) [upload=696432]: Got next segment. Inline: false
Jan 7 19:44:08.333377820 (796h15m8.204460855s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:44:08.333384442 (796h15m8.204467487s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:44:08.588937856 (796h15m8.460020901s) [seg_pos={0 0}, upload=696432]: Begin upload segment...
Jan 7 19:44:08.588994554 (796h15m8.460077589s) [seg_pos={0 0}, upload=696432]: Done begin upload segment.
Jan 7 19:44:08.589025764 (796h15m8.460108809s) [noise=true , address=45.91.116.7:4012 , piece_id=YAOAYOGN6IMTMQJFLFGZXFQJPOS5T5QPOJPQDUH5N2R4GGBY45AQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.589087351 (796h15m8.460170396s) [noise=true , address=145.79.227.15:4873 , piece_id=NIXB5RF73OY52CVGPNHG4X7GYVHZHMSJ5UPGBN2QT2NIJBNEA6QQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.589863268 (796h15m8.460946303s) [noise=true , address=145.79.227.20:4438 , piece_id=45WLNZXX7J7FNUAWTHWZSCVPVZ6ZVTQHPU5XLWGEMXN2ITLAYI7Q, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.590381135 (796h15m8.46146418s) [noise=true , address=145.79.227.35:4482 , piece_id=DCTTZGQIEMP5P2KGRKHXU3SL2LTU4PROWDHP33LT2MLBJZDYVC5Q, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.590896996 (796h15m8.461980041s) [noise=true , address=45.91.116.14:4190 , piece_id=TP2NJXIJXVUKSGLHXI34MFK57UEZZ253EFSAY3QCAJYTE36L3ZRQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.591341222 (796h15m8.462424267s) [noise=true , address=145.79.227.11:4730 , piece_id=4VVGNGW4CKDPBUPKMAHEBASKY2HZ2SEF25C6JUUDTVFU2L3QVSFQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.592060773 (796h15m8.463143818s) [noise=true , address=145.79.227.52:4445 , piece_id=JXL6ZIUQBVVI62FDOQL4WPX4NWCG5YHKRCKDNPLNAG4YWVFXMWZQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.592560794 (796h15m8.46364384s) [noise=true , address=145.79.227.34:4906 , piece_id=2CF2WFERVCOIXVV636BNWNOFTRQ6LRL54UOTRKDZCXBDHDRHASQA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.593282850 (796h15m8.464365895s) [noise=true , address=145.79.227.13:4804 , piece_id=STTU2CXMNNA2FN4JDUNU7T6LWXPICSQUS4G5GZNY7RQ6W4DC67XA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.594933733 (796h15m8.466016778s) [noise=true , address=45.91.116.11:4118 , piece_id=N63NSYUX7FOAQOIGHR7ZAOJT54WROCD46WUAK5V5KM6MBTXMLIDQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.595392266 (796h15m8.466475301s) [noise=true , address=45.91.116.5:4297 , piece_id=VRDCLQSPZDUJB3YQINMQDCYZCASXDKF4MRA2SFXUGXRXGC7UE2YQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.595852353 (796h15m8.466935388s) [noise=true , address=145.79.227.29:4941 , piece_id=EUMSEXARZZLLKIFMU7KNR572KOJY6JKJ4CEXILOZG7EL45HWAZEA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.596283133 (796h15m8.467366178s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2M7NQQMSKVZMFLT4DPIGAJ5V5V3OZV5ENJB7OD55RV7OOWFYBCQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.596691981 (796h15m8.467775026s) [noise=true , address=145.79.227.41:4087 , piece_id=XLT5LPCCTO5YQMXRJVWZ72JQIUNJOZYHBIKC5VFFANKIICBTYFVQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.597184539 (796h15m8.468267584s) [noise=true , address=145.79.227.47:4265 , piece_id=B5PC3HASGN5SU7KBRRPSFINAASBTS5GSYCT5ZYEYP2FGQOHPVXBQ, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:08.597673530 (796h15m8.468756575s) [noise=true , address=45.91.116.4:4939 , piece_id=C4CX7XDCFMA5CKPWQAUHJHCSPL24MUKPCIMDBGJF6GJ5W6JNHEQA, seg_pos={0 0}, upload=696432]: Uploading piece...
Jan 7 19:44:10.202592207 (796h15m10.073675252s) [upload=696432]: Next returned nil segment
Jan 7 19:44:10.202638124 (796h15m10.073721169s) [upload=696432]: Waiting for error group managing segments...
Jan 7 19:44:10.586509756 (796h15m10.457592801s) [noise=true , address=145.79.227.11:4730 , piece_id=4VVGNGW4CKDPBUPKMAHEBASKY2HZ2SEF25C6JUUDTVFU2L3QVSFQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.589115648 (796h15m10.460198693s) [noise=true , address=145.79.227.15:4873 , piece_id=NIXB5RF73OY52CVGPNHG4X7GYVHZHMSJ5UPGBN2QT2NIJBNEA6QQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.589194348 (796h15m10.460277393s) [noise=true , address=145.79.227.35:4482 , piece_id=DCTTZGQIEMP5P2KGRKHXU3SL2LTU4PROWDHP33LT2MLBJZDYVC5Q, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.594279190 (796h15m10.465362235s) [noise=true , address=145.79.227.47:4265 , piece_id=B5PC3HASGN5SU7KBRRPSFINAASBTS5GSYCT5ZYEYP2FGQOHPVXBQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.597989425 (796h15m10.46907247s) [noise=true , address=145.79.227.20:4438 , piece_id=45WLNZXX7J7FNUAWTHWZSCVPVZ6ZVTQHPU5XLWGEMXN2ITLAYI7Q, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.601834046 (796h15m10.472917081s) [noise=true , address=145.79.227.52:4445 , piece_id=JXL6ZIUQBVVI62FDOQL4WPX4NWCG5YHKRCKDNPLNAG4YWVFXMWZQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.603204615 (796h15m10.47428766s) [noise=true , address=145.79.227.41:4087 , piece_id=XLT5LPCCTO5YQMXRJVWZ72JQIUNJOZYHBIKC5VFFANKIICBTYFVQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.607187650 (796h15m10.478270695s) [noise=true , address=145.79.227.34:4906 , piece_id=2CF2WFERVCOIXVV636BNWNOFTRQ6LRL54UOTRKDZCXBDHDRHASQA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.658088834 (796h15m10.529171869s) [noise=true , address=145.79.227.13:4804 , piece_id=STTU2CXMNNA2FN4JDUNU7T6LWXPICSQUS4G5GZNY7RQ6W4DC67XA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:10.987236272 (796h15m10.858319307s) [noise=true , address=45.91.116.11:4118 , piece_id=N63NSYUX7FOAQOIGHR7ZAOJT54WROCD46WUAK5V5KM6MBTXMLIDQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.007251394 (796h15m10.878334459s) [noise=true , address=45.91.116.14:4190 , piece_id=TP2NJXIJXVUKSGLHXI34MFK57UEZZ253EFSAY3QCAJYTE36L3ZRQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.152638965 (796h15m11.02372201s) [noise=true , address=45.91.116.7:4012 , piece_id=YAOAYOGN6IMTMQJFLFGZXFQJPOS5T5QPOJPQDUH5N2R4GGBY45AQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.195362016 (796h15m11.066445061s) [noise=true , address=45.91.116.4:4939 , piece_id=C4CX7XDCFMA5CKPWQAUHJHCSPL24MUKPCIMDBGJF6GJ5W6JNHEQA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: <nil>
Jan 7 19:44:11.195383697 (796h15m11.066466732s) [seg_pos={0 0}, upload=696432]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:44:11.195634255 (796h15m11.0667173s) [noise=true , address=145.79.227.29:4941 , piece_id=EUMSEXARZZLLKIFMU7KNR572KOJY6JKJ4CEXILOZG7EL45HWAZEA, seg_pos={0 0}, upload=696432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Jan 7 19:44:11.195642560 (796h15m11.066725606s) [noise=true , address=145.79.227.29:4941 , piece_id=EUMSEXARZZLLKIFMU7KNR572KOJY6JKJ4CEXILOZG7EL45HWAZEA, seg_pos={0 0}, upload=696432]: Long tail cancellation due to optimal threshold
Jan 7 19:44:11.195899650 (796h15m11.066982685s) [noise=true , address=45.91.116.5:4297 , piece_id=VRDCLQSPZDUJB3YQINMQDCYZCASXDKF4MRA2SFXUGXRXGC7UE2YQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Jan 7 19:44:11.195909068 (796h15m11.066992103s) [noise=true , address=45.91.116.5:4297 , piece_id=VRDCLQSPZDUJB3YQINMQDCYZCASXDKF4MRA2SFXUGXRXGC7UE2YQ, seg_pos={0 0}, upload=696432]: Long tail cancellation due to optimal threshold
Jan 7 19:44:11.195924257 (796h15m11.067007312s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2M7NQQMSKVZMFLT4DPIGAJ5V5V3OZV5ENJB7OD55RV7OOWFYBCQ, seg_pos={0 0}, upload=696432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Jan 7 19:44:11.195945958 (796h15m11.067029004s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2M7NQQMSKVZMFLT4DPIGAJ5V5V3OZV5ENJB7OD55RV7OOWFYBCQ, seg_pos={0 0}, upload=696432]: Long tail cancellation due to optimal threshold
Jan 7 19:44:11.195980685 (796h15m11.06706373s) [seg_pos={0 0}, upload=696432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:44:11.196077739 (796h15m11.067160784s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:44:11.196084372 (796h15m11.067167417s) [upload=696432]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:44:11.480318939 (796h15m11.351401984s) [upload=696432]: Done uploading segments...
Jan 7 19:44:11.480349377 (796h15m11.351432412s) [upload=696432]: Upload finished. err: <nil>
Duration: 30.002510163s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
28 out of 60 is failed (threshold=6) [X.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X......] duration:
3.484068802s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:44:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.484096044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
29 out of 60 is failed (threshold=6) [XX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X.....] duration:
4.244025771s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:43:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.244053974s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
259.942841ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:43:33 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 259.970434ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X..X..X.X.X....X........X..........X..XX..X.X....XX......] duration:
6.134414347s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:43:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 6.134433291s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
30 out of 60 is failed (threshold=6) [XXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X....] duration:
3.630606265s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:43:21 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.630628168s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X..X..X.X.X....X........X..........X..XX..X.X....XX.....] duration:
6.409721324s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:43:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 6.409752082s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
270.52401ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:42:30 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 270.548798ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [XXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X...] duration:
3.576917959s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:42:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.576938428s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....X..X..X.X.X....X........X..........X..XX..X.X....XX....] duration:
5.692076944s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:41:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 5.692099076s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
261.873077ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:41:30 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 261.8956ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X.....X..X..X.X.X....X........X..........X..XX..X.X....XX...] duration:
5.838091329s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:41:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 5.838117038s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [XXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X..] duration:
3.558271787s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:41:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.558297857s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [XXXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X.] duration:
3.632286541s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:40:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.632316879s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
259.151955ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:40:50 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 259.18033ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.306890556s trace:
https://jaeger.dev.storj.io/trace/6795275e4d2bb888
FAIL (2026-01-07 19:40:28 UTC)
orbiter warning: context deadline exceeded
Duration: 6.306925453s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [XXXXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...X] duration:
5.001774541s trace:
https://jaeger.dev.storj.io/trace/6e75a96768548c4
FAIL (2026-01-07 19:40:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001796284s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
273.386811ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:40:20 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 273.419144ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X.....X..X..X.X.X....X........X..........X..XX..X.X....XX..] duration:
6.052516638s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:40:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 6.052540823s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.015101784s trace:
https://jaeger.dev.storj.io/trace/6795275e4d2bb888
FAIL (2026-01-07 19:39:44 UTC)
orbiter warning: context deadline exceeded
Duration: 6.015128105s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
264.655856ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:39:37 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 264.679651ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [XXXXXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX...] duration:
3.504895464s trace:
https://jaeger.dev.storj.io/trace/269712876405a49b
FAIL (2026-01-07 19:39:30 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.504955128s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X.....X..X..X.X.X....X........X..........X..XX..X.X....XX.] duration:
5.972605367s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:39:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 5.972640595s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [.XXXXXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX..] duration:
3.463380358s trace:
https://jaeger.dev.storj.io/trace/269712876405a49b
FAIL (2026-01-07 19:38:53 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.463407209s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
7.918703317s trace:
https://jaeger.dev.storj.io/trace/6795275e4d2bb888
FAIL (2026-01-07 19:38:50 UTC)
orbiter warning: context deadline exceeded
Duration: 7.918732963s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...X.....X..X..X.X.X....X........X..........X..XX..X.X....XX] duration:
30.002989225s trace:
https://jaeger.dev.storj.io/trace/10127be6f351cb12
FAIL (2026-01-07 19:38:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 19:38:20.096266592 (796h9m19.967349627s) [upload=696341]: Starting upload
Jan 7 19:38:20.096348087 (796h9m19.967431122s) [upload=696341]: Done starting upload
Jan 7 19:38:20.096418361 (796h9m19.967501406s) [upload=696341]: Uploading segments...
Jan 7 19:38:20.096510627 (796h9m19.967593662s) [upload=696341]: Got next segment. Inline: false
Jan 7 19:38:20.096538480 (796h9m19.967621515s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:38:20.096542738 (796h9m19.967625783s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:38:20.349753202 (796h9m20.220836247s) [seg_pos={0 0}, upload=696341]: Begin upload segment...
Jan 7 19:38:20.349826051 (796h9m20.220909126s) [seg_pos={0 0}, upload=696341]: Done begin upload segment.
Jan 7 19:38:20.349859835 (796h9m20.22094288s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349886566 (796h9m20.220969602s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.349945508 (796h9m20.221028543s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350602269 (796h9m20.221685304s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.350978936 (796h9m20.222061981s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351378617 (796h9m20.222461662s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.351754513 (796h9m20.222837548s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352178570 (796h9m20.223261605s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.352609181 (796h9m20.223692216s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353060540 (796h9m20.224143605s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.353633300 (796h9m20.224716345s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.354888260 (796h9m20.225971295s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.356451647 (796h9m20.227534682s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.357866261 (796h9m20.228949296s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.360536205 (796h9m20.23161925s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:20.363819056 (796h9m20.234902091s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Uploading piece...
Jan 7 19:38:22.060957818 (796h9m21.932040863s) [upload=696341]: Next returned nil segment
Jan 7 19:38:22.060975651 (796h9m21.932058697s) [upload=696341]: Waiting for error group managing segments...
Jan 7 19:38:22.343010713 (796h9m22.214093748s) [noise=true , address=145.79.227.6:4257 , piece_id=TEKBIMGSS6HNLENP6XNTXRMH5ANOW4VMXAOTIOFBFPZDM433VRDA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.345424709 (796h9m22.216507744s) [noise=true , address=145.79.227.37:4567 , piece_id=77VLOUVBFVCLR7ZKPRV7Z5X6ZZRNTKBO25I7P6XYWVKOHTKGMJVA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.358098071 (796h9m22.229181107s) [noise=true , address=145.79.227.42:4123 , piece_id=Y7UQ5SBRHI3NQJWE4XSAPVHSHVED5ECU5BRYFE2KNYJKWM6N2YMA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.368687667 (796h9m22.239770702s) [noise=true , address=145.79.227.23:4546 , piece_id=4UW2P27UP4E7XDEK7WZQ7YQQZPXJOYWSUGS5AGIQYCSWPFQ3MMPA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.373487966 (796h9m22.244571011s) [noise=true , address=145.79.227.4:4186 , piece_id=6JKVIMN3EGBHCTDHW2APBBKZNAINM2POSNSE6OEDDC5IGNUHVHOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.379599484 (796h9m22.250682519s) [noise=true , address=145.79.227.16:4909 , piece_id=L4E7EDWY5K6ONPCFAIS2UXFD7DVVMUTLK2OZXGC7QTQ6MUJXRXSQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.428457448 (796h9m22.299540493s) [noise=true , address=145.79.227.44:4195 , piece_id=MDEWZX5RNQ7WSCDTU7AVJGV2ZBZ6Z2GNKVVTL5T7NCEZCERXAO2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.454319894 (796h9m22.325402939s) [noise=true , address=145.79.227.41:4086 , piece_id=HNLULLJGVM45RIE534562BX6ABAXN3IUGZRDL64YRTMUIKYDSH2A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.463556292 (796h9m22.334639337s) [noise=true , address=145.79.227.14:4837 , piece_id=DGTJA5ALJBNVTCHXV5ETAKLQVII7JHTIVB3TFSSQIYQXG6MXV3NQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.651909928 (796h9m22.522992973s) [noise=true , address=77.90.31.15:4700 , piece_id=7UDSFZAX77T2MBHTNDDNAV7DXJ2YHGJUVKGIC3YTCFJLR62XS3IA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:22.841067255 (796h9m22.71215029s) [noise=true , address=45.91.116.9:4261 , piece_id=UH74TKFWLOTR43JQDAG4EYOAPEFYWIV3V2OY6FC5PETJN5FAEPHA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.231717829 (796h9m23.102800874s) [noise=true , address=45.91.116.4:4939 , piece_id=5IS47XOFMQQFBF47PQABP5OVHUZVPM3342SVPKCNGRZOHAHACW3A, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857599712 (796h9m23.728682757s) [noise=true , address=77.90.31.16:4736 , piece_id=2SR5UL4DLNHDZLWD3IDVTN2KTNAZYGJAEGXRS4TKFTS2A73JM6PQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: <nil>
Jan 7 19:38:23.857625421 (796h9m23.728708466s) [seg_pos={0 0}, upload=696341]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:38:23.857890906 (796h9m23.728973951s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857902267 (796h9m23.728985322s) [noise=true , address=145.79.227.26:4655 , piece_id=E2UB3TLK4M5BZPJDWCGFA6JEFCN72IRBXRIDGRQSGPY3LP3CWODA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.857956771 (796h9m23.729039826s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.857967321 (796h9m23.729050366s) [noise=true , address=77.90.31.21:4878 , piece_id=6VHAVVUDY4ZZIEGUJSIPZAN7QGUCV5PPEIUCZAD75VZSJH3YCRAA, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858159227 (796h9m23.729242282s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF
Jan 7 19:38:23.858167933 (796h9m23.729250968s) [noise=true , address=145.79.227.21:4473 , piece_id=I7XHT5MTJNGAVVDKBECOOLBKM6ULWV6OVOBDRTFK3TW5ULVXYIOQ, seg_pos={0 0}, upload=696341]: Long tail cancellation due to optimal threshold
Jan 7 19:38:23.858200205 (796h9m23.72928326s) [seg_pos={0 0}, upload=696341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:38:23.858292220 (796h9m23.729375265s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:38:23.858298242 (796h9m23.729381277s) [upload=696341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:38:24.147197574 (796h9m24.018280609s) [upload=696341]: Done uploading segments...
Jan 7 19:38:24.147225927 (796h9m24.018308972s) [upload=696341]: Upload finished. err: <nil>
Duration: 30.003005104s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
283.860758ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:38:29 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 283.885826ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [X.XXXXXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX.] duration:
3.524130247s trace:
https://jaeger.dev.storj.io/trace/269712876405a49b
FAIL (2026-01-07 19:38:23 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.524157788s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X.....X..X..X.X.X....X........X..........X..XX..X.X....X] duration:
30.003573277s trace:
https://jaeger.dev.storj.io/trace/30fe357b0e2a9106
FAIL (2026-01-07 19:38:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: P2Z7C5YSQKOR3KL2L6HYUPJFITQQDZCXTPV4JKJHSSUBD6L3QMKA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: KL66337XCZCJFTM7WQRC7U2LHFBWAKJCVKP53NT74JZLOYUGE6DA) context deadline exceeded
======LOGS======
Jan 7 19:37:40.894063044 (796h8m40.765146089s) [upload=696331]: Starting upload
Jan 7 19:37:40.894144018 (796h8m40.765227063s) [upload=696331]: Done starting upload
Jan 7 19:37:40.894309623 (796h8m40.765392668s) [upload=696331]: Uploading segments...
Jan 7 19:37:40.894454048 (796h8m40.765537093s) [upload=696331]: Got next segment. Inline: false
Jan 7 19:37:40.894493643 (796h8m40.765576688s) [upload=696331]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 19:37:40.894499705 (796h8m40.76558275s) [upload=696331]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 19:37:41.155426342 (796h8m41.026509388s) [seg_pos={0 0}, upload=696331]: Begin upload segment...
Jan 7 19:37:41.155488301 (796h8m41.026571336s) [seg_pos={0 0}, upload=696331]: Done begin upload segment.
Jan 7 19:37:41.155528968 (796h8m41.026612013s) [noise=true , address=145.79.227.45:4231 , piece_id=DYZK2ST2SYQVGONZTLJZ36BR5EIV2UXMMWQGJ4ZI3L7S4PRJMU6Q, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.155654957 (796h8m41.026737992s) [noise=true , address=145.79.227.37:4565 , piece_id=BJJUSKO2CFLOH63HDCOCYT4F5VPTYOY2PW2OLPFIK4ZER5H6NSZQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.156276621 (796h8m41.027359656s) [noise=true , address=145.79.227.16:4909 , piece_id=LSZ4BHA4CKX42UGFDWKSRIRRJYSEA2VI7BOWOQIKG3IDZCQ5RUSA, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.156415536 (796h8m41.027498581s) [noise=true , address=145.79.227.10:4403 , piece_id=A7TDDMPYVSC6QBREMAQTATOJ62QYLROTQXRGKIBHMARYEEAOMY3A, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.156622490 (796h8m41.027705525s) [noise=true , address=77.90.31.11:4555 , piece_id=KL66337XCZCJFTM7WQRC7U2LHFBWAKJCVKP53NT74JZLOYUGE6DA, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.156812803 (796h8m41.027895848s) [noise=true , address=145.79.227.31:4799 , piece_id=VVBEVL3PJNADQD6P4IM6KIFAO64RV6LU5Z3POIHGFK4EQL67HNTQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.156959983 (796h8m41.028043028s) [noise=true , address=45.91.116.10:4083 , piece_id=SUMZZY5E3PKY3JGJSEDWWJEAJAQURU33RHOODWABINQLMW3UBPXQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.157209257 (796h8m41.028292302s) [noise=true , address=45.91.116.14:4192 , piece_id=GNJKLLS2XM3GSE6OQZYRHYSZEFJNWUI6KO6WCBDRTR4VOZIXBHIQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.157340888 (796h8m41.028423923s) [noise=true , address=45.91.116.6:4974 , piece_id=P2Z7C5YSQKOR3KL2L6HYUPJFITQQDZCXTPV4JKJHSSUBD6L3QMKA, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.157636741 (796h8m41.028719776s) [noise=true , address=145.79.227.6:4259 , piece_id=EDL2YH7XYGLJJUDXUFI62EUDUJQE7QFCGN5XI3BDQBBOQAFUZG3A, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.157762491 (796h8m41.028845526s) [noise=true , address=145.79.227.5:4223 , piece_id=MRFEMKJE6EKKDPQLDASM3FJVYFVOYNL2ATTYIGLFSMYJVMLPPYYA, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.158016654 (796h8m41.0290997s) [noise=true , address=145.79.227.41:4085 , piece_id=326N5TN3YRIUNGAFAUMMVBKWUXIMTAB4VZ2CWGEVGSIOSTB3H5IQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.158112417 (796h8m41.029195452s) [noise=true , address=145.79.227.24:4582 , piece_id=2MHKZDV4KS6NXLZ7ZCE5XQNK2B7GONEMBNS2LMQ6DLIIELAJTHKQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.158359558 (796h8m41.029442593s) [noise=true , address=145.79.227.40:4051 , piece_id=XD6LXJ2H6EXE3SGBNBBKRJHLTGOYVTIHCAUDSJGDWUAEUPZOMSYA, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.158469447 (796h8m41.029552492s) [noise=true , address=45.91.116.7:4011 , piece_id=ITYPMZAMHJLFXQBGJIVYJZUUPEUELTAFAK5BPKFXLVBLE7SCYGYQ, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:41.158653578 (796h8m41.029736623s) [noise=true , address=145.79.227.15:4873 , piece_id=BORG5AZIBPQK4G3G3MRFYJOJWRGVXH6JMGDXWGC4NSEUPHIATLKA, seg_pos={0 0}, upload=696331]: Uploading piece...
Jan 7 19:37:42.981864864 (796h8m42.852947909s) [upload=696331]: Next returned nil segment
Jan 7 19:37:42.981880745 (796h8m42.85296379s) [upload=696331]: Waiting for error group managing segments...
Jan 7 19:37:43.284353833 (796h8m43.155436868s) [noise=true , address=145.79.227.16:4909 , piece_id=LSZ4BHA4CKX42UGFDWKSRIRRJYSEA2VI7BOWOQIKG3IDZCQ5RUSA, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.317113433 (796h8m43.188196468s) [noise=true , address=145.79.227.5:4223 , piece_id=MRFEMKJE6EKKDPQLDASM3FJVYFVOYNL2ATTYIGLFSMYJVMLPPYYA, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.339240766 (796h8m43.210323811s) [noise=true , address=145.79.227.45:4231 , piece_id=DYZK2ST2SYQVGONZTLJZ36BR5EIV2UXMMWQGJ4ZI3L7S4PRJMU6Q, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.365762729 (796h8m43.236845764s) [noise=true , address=145.79.227.24:4582 , piece_id=2MHKZDV4KS6NXLZ7ZCE5XQNK2B7GONEMBNS2LMQ6DLIIELAJTHKQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.375232130 (796h8m43.246315175s) [noise=true , address=145.79.227.10:4403 , piece_id=A7TDDMPYVSC6QBREMAQTATOJ62QYLROTQXRGKIBHMARYEEAOMY3A, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.466696323 (796h8m43.337779358s) [noise=true , address=145.79.227.40:4051 , piece_id=XD6LXJ2H6EXE3SGBNBBKRJHLTGOYVTIHCAUDSJGDWUAEUPZOMSYA, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.497336868 (796h8m43.368419913s) [noise=true , address=77.90.31.11:4555 , piece_id=KL66337XCZCJFTM7WQRC7U2LHFBWAKJCVKP53NT74JZLOYUGE6DA, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.660382924 (796h8m43.531465959s) [noise=true , address=45.91.116.14:4192 , piece_id=GNJKLLS2XM3GSE6OQZYRHYSZEFJNWUI6KO6WCBDRTR4VOZIXBHIQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:43.735065582 (796h8m43.606148617s) [noise=true , address=145.79.227.6:4259 , piece_id=EDL2YH7XYGLJJUDXUFI62EUDUJQE7QFCGN5XI3BDQBBOQAFUZG3A, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:44.038835831 (796h8m43.909918866s) [noise=true , address=145.79.227.15:4873 , piece_id=BORG5AZIBPQK4G3G3MRFYJOJWRGVXH6JMGDXWGC4NSEUPHIATLKA, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:44.090140964 (796h8m43.961224009s) [noise=true , address=145.79.227.31:4799 , piece_id=VVBEVL3PJNADQD6P4IM6KIFAO64RV6LU5Z3POIHGFK4EQL67HNTQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:44.213857171 (796h8m44.084940206s) [noise=true , address=145.79.227.41:4085 , piece_id=326N5TN3YRIUNGAFAUMMVBKWUXIMTAB4VZ2CWGEVGSIOSTB3H5IQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:44.412264191 (796h8m44.283347226s) [noise=true , address=45.91.116.6:4974 , piece_id=P2Z7C5YSQKOR3KL2L6HYUPJFITQQDZCXTPV4JKJHSSUBD6L3QMKA, seg_pos={0 0}, upload=696331]: Done uploading piece. err: <nil>
Jan 7 19:37:44.412284148 (796h8m44.283367193s) [seg_pos={0 0}, upload=696331]: Segment reached optimal threshold of 13 pieces.
Jan 7 19:37:44.412460093 (796h8m44.283543138s) [noise=true , address=45.91.116.10:4083 , piece_id=SUMZZY5E3PKY3JGJSEDWWJEAJAQURU33RHOODWABINQLMW3UBPXQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 19:37:44.412466075 (796h8m44.28354911s) [noise=true , address=45.91.116.10:4083 , piece_id=SUMZZY5E3PKY3JGJSEDWWJEAJAQURU33RHOODWABINQLMW3UBPXQ, seg_pos={0 0}, upload=696331]: Long tail cancellation due to optimal threshold
Jan 7 19:37:44.412472638 (796h8m44.283555673s) [noise=true , address=145.79.227.37:4565 , piece_id=BJJUSKO2CFLOH63HDCOCYT4F5VPTYOY2PW2OLPFIK4ZER5H6NSZQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF
Jan 7 19:37:44.412480883 (796h8m44.283563918s) [noise=true , address=145.79.227.37:4565 , piece_id=BJJUSKO2CFLOH63HDCOCYT4F5VPTYOY2PW2OLPFIK4ZER5H6NSZQ, seg_pos={0 0}, upload=696331]: Long tail cancellation due to optimal threshold
Jan 7 19:37:44.412572378 (796h8m44.283655423s) [noise=true , address=45.91.116.7:4011 , piece_id=ITYPMZAMHJLFXQBGJIVYJZUUPEUELTAFAK5BPKFXLVBLE7SCYGYQ, seg_pos={0 0}, upload=696331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Jan 7 19:37:44.412577487 (796h8m44.283660522s) [noise=true , address=45.91.116.7:4011 , piece_id=ITYPMZAMHJLFXQBGJIVYJZUUPEUELTAFAK5BPKFXLVBLE7SCYGYQ, seg_pos={0 0}, upload=696331]: Long tail cancellation due to optimal threshold
Jan 7 19:37:44.412593076 (796h8m44.283676111s) [seg_pos={0 0}, upload=696331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 19:37:44.412654624 (796h8m44.283737659s) [upload=696331]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 19:37:44.412658882 (796h8m44.283741917s) [upload=696331]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 19:37:45.038894237 (796h8m44.909977272s) [upload=696331]: Done uploading segments...
Jan 7 19:37:45.038912853 (796h8m44.909995888s) [upload=696331]: Upload finished. err: <nil>
Duration: 30.003593496s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
9.661572007s trace:
https://jaeger.dev.storj.io/trace/6795275e4d2bb888
FAIL (2026-01-07 19:37:57 UTC)
orbiter warning: context deadline exceeded
Duration: 9.661587406s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
270.751426ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:37:56 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 270.777345ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
32.564298192s remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/31bb9f6974376f26
FAIL (2026-01-07 19:37:51 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 32.564319342s
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XXXX..XXXX.X.XXX....XXXX..XXXX.X..X...X....X..XX] duration:
5.001224588s trace:
https://jaeger.dev.storj.io/trace/269712876405a49b
FAIL (2026-01-07 19:37:39 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.001239316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
12.017722593s trace:
https://jaeger.dev.storj.io/trace/6795275e4d2bb888
FAIL (2026-01-07 19:37:21 UTC)
orbiter warning: context deadline exceeded
Duration: 12.017748301s
edge
gateway
us-select-1
|